Joseph Wilkinshttps://josephwilkins.wordpress.com
Notes on interaction designWed, 05 Apr 2017 18:49:51 +0000enhourly1http://wordpress.com/https://s2.wp.com/i/buttonw-com.pngJoseph Wilkinshttps://josephwilkins.wordpress.com
Interactive Hologramhttps://josephwilkins.wordpress.com/2015/04/04/gestural-interactive-hologram/
https://josephwilkins.wordpress.com/2015/04/04/gestural-interactive-hologram/#respondSat, 04 Apr 2015 10:47:00 +0000http://josephwilkins.wordpress.com/?p=577]]>One of my hang ups with the Leap Motion Controller is that there is a shortage of options when it comes to effective 3D visual outputs which take full advantage of the added dimension of control. Traditional 2D screens, or even projections, don’t accurately convey the sense of control over depth that the Leap provides. This is something that I’ve alluded to in a previous post regarding some of the possible applications of the Leap Motion Controller.

Last summer, I created a basic interactive product demo at 9 Yards, which utilised the DreamOC hologram unit and a Leap motion controller, to demonstrate a possible concept which allows the Leap to be used to create an interactive product catalogue. The result of which can be seen in the video below.

After showing this demo to a few clients, one in particular, a lightbulb manufacturer, became really interested in the idea and we worked with them to develop the initial concept into a full application. The idea was to turn the application into a holographic catalogue which would allow users to browse, select and view different types of lamp.

We took the initial concept, and proposed an application which would span a shared desktop over two HD resolution screens, the first being a holographic display and the second being a touch small OEM screen, a Leap Motion Controller would be mounted above. The application functionality would provide the user with an interface with which to navigate through a catalogue of lamps, using a 2D GUI on a touch screen mounted below the hologram. When a lamp was selected it would appear in the hologram, allowing the user to rotate it using gestures.

Whilst the initial demo was created using Flash and the Papervision3D library, I developed the final application in Unity3D which is a far more suitable and capable interactive 3D development environment. This was my first major project using Unity and it proved to be really powerful, allowing for really easy integration with the Leap Motion Controller as an input device, SQLite database functionality to track metrics on interaction and customer details, and most important, excellent real-time rendering and display of 3D models with really effective lighting and texturing.

We hit a few challenges with development along the way, here are a few of the most notable, and worth bearing in mind:

– Models:

Modelling can be done with pretty much any modelling software, Cinema 4D, Maya, Blender and so on. However, the tricky and sometimes unpredictable aspect of the modelling for us was the point at which you import into Unity. Often I would have to reapply materials after import in order to ensure that the lamps appeared as intended. To some extent, this was also true of the geometries within models – occasionally, we found that models would import into Unity in an incomplete state, this required us to make a few touch ups once they were in Unity to correct and missing features. I expect this may also be remedied through further investigation in to the export settings in the modelling environment. It’s also worth noting that Marmoset’s skyshop shaders came in really handy for this project!

– Dividing the viewport:

In order to show the lamp model on the holographic display, we had to create three separate viewports and render a different camera angle into each. This is due to the way in which the DreamOC hologram unit creates the holographic effect. Three angled two-way mirrors reflect an image from a screen mounted above them, this meant that the setup for the hologram view of the app was effectively three masked viewports which displayed the front and side views of the lamp model – it took quite a bit of tweaking to get the right effect!

– Unity updates:

Perhaps one of the most irritating issues I faced whilst developing with Unity was that, during the development cycle, updates to the software were released. On a few occasions, I went ahead with updates, not expecting them to compromise the stability of the application in development. Unfortunately, one of the updates, the major version change from version 4 to 5 caused a lot of issues, due to updates to the API. With hindsight, it’s probably a good idea to proceed with caution if you’re considering major software updates whilst heavily involved with a project. Luckily, that’s one of the many things that version control is useful for!

Here’s a short clip of the completed hologram element of the application which demonstrates a bulb being selected for viewing.

It was a great project to develop and I think the end result was really effective, it represented a real deep dive into Unity for me and I really enjoyed it. I’ll definitely be working with Unity in future and I’m really interested in the possible applications of 3D game engines, particularly outside of the realm of gaming, I think there’s a lot that can be done, especially when combining digital visuals with physical inputs and outputs.

]]>https://josephwilkins.wordpress.com/2015/04/04/gestural-interactive-hologram/feed/0carousel_1_imagejosephwilkinsInteractive hologram in situWorking with Oculus Rift DK2https://josephwilkins.wordpress.com/2015/03/08/working-with-oculus-rift-dk2/
https://josephwilkins.wordpress.com/2015/03/08/working-with-oculus-rift-dk2/#respondSun, 08 Mar 2015 11:07:00 +0000http://josephwilkins.wordpress.com/?p=602]]>Since it’s initial Kickstarter campaign in 2012, the Oculus Rift Virtual Reality (VR) headset has reignited interest in the concept of virtual reality as medium for immersive entertainment. As a kid, I vaguely remember VR headsets making a brief appearance on technology based TV programmes, though the trend seemed to come and go, and certainly never made it into the home. It’s latest incarnation, more than 20 years later, has come largely thanks to the rise of smart phones. Technology has developed so significantly that we are now in a position where everything from screen resolution, processing power and developments in spatial and motion tracking, have come far enough to allow us to compress most of the technology required for VR into a lighter weight headset that can be worn, I imagine, far more comfortably than those that were developed during the early 90’s.

At 9 yards we spent some time looking into the possibilities for the development of VR experiences, and as Application Developer, I prototyped a few rough concepts to demonstrate ways in which we could use the medium.

Being a format which is fundamentally rooted in the exploration of three dimensional environments, we opted to use a 3D development platform to create content to use with the Oculus Rift. Here, Unity was a natural choice as integration with the Oculus Rift is easily achieved using a prefab of the head set, which has cameras positioned correctly for perspective, among a number of games development based features which lend themselves well to VR development.

With a large number of our clients existing in the medicine and pharmaceuticals industries, our initial “Hello World” type VR application was a virtual journey through a bloodstream, in which the various cells where represented in a larger abstract form to demonstrate the movement of blood though the veins. Within this demo, we quickly realised how movement and perception of space are critical considerations when developing for VR. In particular, we noticed that there is a significant disconnect between the viewer’s real world movement and the movement being portrayed within the interactive experience. Getting this balance right is an essential part of the development of experiences for VR and can make the difference between content which is enjoyable to engage with and that which is physically uncomfortable to use.

In the demo, the viewer simply moves through the blood stream on a fixed track whilst red blood cells and other types of cell representing a virus also fly past at different speeds. It’s easy to see how VR can be used to create any number of real life or fantasy worlds and indeed, explore what it might be like to have alternative perspectives on situations or occurrences which you may not otherwise have access to.

Beyond this simple VR experience, I wanted to look at other ways in which the Oculus Rift could be used. One of my own personal reservations about the Oculus Rift, and indeed the many other forthcoming VR headsets, is that they seem to be leading us towards a future of entertainment technologies which create exclusive and self contained experiences. As a technology, VR is entirely reliant on using one person’s point of view as the perspective from which to manipulate a scene. It is therefore inherently a single user format in which only the wearer of the headset can become fully immersed. This, to a large extent, prevents or at very least reduces the ability of the format to allow for experiences which are inclusive, collaborative and which bring people together.

With this in mind, I thought it would be interesting to look at ways in which use of VR could be turned into a more collaborative or inclusive experience allowing more than one person to take part in the same experience. I thought that the multi player format would be a good way to provide an environment in which several people could exist within a VR world and enjoy the same content, whilst also maintaining their own personal view point. I developed a prototype in which multiple VR headsets can be used to drop into the same room. Each viewer can see one another whilst also being able to see the same objects within the room, consequently, each person can manipulate the objects and indeed see each other manipulating objects.

In this more abstract prototype, each person within the multiplayer VR experience is represented as a coloured cube. Within the room, there are coloured balls on the ground which can be pushed around by the movement of the player, and high within the room there is a model of planet earth which rotates. These simple interactive and non interactive elements were placed within the room test perspective and interaction between multiple players, kind of like a Cornell Box for VR interactivity. Within this prototype, it is possible to see how VR experience could be used to create inclusive and collaborative experiences, if developed as a multiuser experience.

Building these basic prototypes allowed us to gain a better insight into some of the challenges involved with developing VR experiences and the sorts of considerations we should make in future VR projects and hopefully we’ll get the opportunity to develop further projects for VR.

At present it feels as though VR, as a form of entertainment, is very much in it’s infancy but there is almost certainly a much stronger proposition in this iteration, when compared with previous attempts at bringing the technology to the mainstream. A number of companies are investing in research and development of VR technologies including Samsung, Microsoft and HTC which indicates that it is being taken more seriously as a medium.

Further to this, it’s really interesting to see the way in which Leap Motion, who’s motion control device I have written about previously, are really pushing their device as a peripheral for VR headsets. I think this is a particulary interesting move as it seems that so far, VR hardware being shipped consist mainly of a head mounted display with no appropriate means of input, other than what is attached to the computer being used. In most cases, this is typically a keyboard and mouse and these aren’t particularly well designed for scenarios in which your perception of space is altered and you can’t see them in your peripheral field of view.

It’s also worth noting that we’re just beginning to see some more interesting use cases for VR, beyond the obvious game based content. Oculus Story Studio, for example, is a seperate division of Oculus VR, setup for the purpose of virtual reality film making and I expect we’ll see more of this in future. Its first film ‘Lost’ looks really nice and I’m excited to see how this form of immersive entertainment develops.

I think the next logical step will be to see how peripherals could be used to enhance both VR, and its close relative, Augmented Reality (AR), both as a means of input and as a method of providing feedback by altering the environment of, and further enhancing the experience of the viewer. Microsoft Research’s IllumiRoom and Simon de Diesbach’s OccultUs present two contrasting but really exciting concepts of how VR and AR could provide truly captivating multi sensory experiences and I think it would really exciting to see VR develop like this, in a more inclusive and immersive way.

]]>https://josephwilkins.wordpress.com/2015/03/08/working-with-oculus-rift-dk2/feed/0josephwilkinsOculus Rift DK2 Blood stream demoOculus Rift DK2 Multiuser demoOculus Rift DK2 Multiuser demoThe chameleon boxhttps://josephwilkins.wordpress.com/2015/01/17/the-chameleon-box/
https://josephwilkins.wordpress.com/2015/01/17/the-chameleon-box/#respondSat, 17 Jan 2015 16:50:51 +0000http://josephwilkins.wordpress.com/?p=498]]>Late last year, we had an in house brief at 9 Yards to come up with ideas for items, objects or installations to be displayed in the alcoves of the feature wall in our office.

My response to the brief was to turn one of the alcoves into a colour sensing light box – place an object within the box, and the lighting in the box would change to match the colour of that object. I worked with various computer vision techniques last year and thought that this would be a good challenge to combine and experiment with some of the techniques that I’ve picked up including background subtraction and colour averaging.

My other aim with the project was to experiment a bit more with the Raspberry Pi. I’ve had one for a while now and have found that you can do some really cool stuff with it, especially when combined with other hardware like the camera module and an Arduino. I also wanted a good excuse to dabble with the Python programming language as I hadn’t had much exposure to it until getting a Raspberry Pi.

There are a few key elements that make up the functionality of the box, the image capture, made possible using the Raspberry Pi camera module, the background subtraction and colour averaging processes made available through the Python CV2 library, and the lighting, which takes the form of a NeoPixel addressable RGB LED ring attached to an Arduino.

First off, the camera module. The Raspberry Pi camera module is a really nice addition to the device which enables stills or video to be captured, there’s even a Pi NoIR camera module which allows for infrared light to be captured, making things like night vision possible, when combined with a infrared light source. The module is really easy to install, and there’s a Python library that you can use to operate the different features. For the purposes of the colour averaging functionality, I needed to capture a series of photos, the first, would be a reference photo – taken of the empty box, illuminated by a neutral white light. This image is then stored and combined with the captured image, which features the object within the box, to create a difference mask which is then used to extract the background of the original captured image. Next is the colour averaging process, the captured image, minus the background is processed using the CV2 library to determine the average colour of the masked pixels. I also added the ability for the reference image to be recaptured to compensate for changes in ambient light conditions. The opening of the box is relatively close to a window and throughout the course of the day the brightness of the light changes as the sun passes over the office. Additionally, as the sunlight fades, the temperature of the lighting changes from a blue tinted outdoor light, to the more yellow tinted incandescent light of the office. See images of the different stages of colour averaging process below.

Image – from top left to bottom right: reference image, captured scene, difference mask, background subtraction. After establishing the average colour of the scene, it’s time to send a hex value to the light source in order to display the matched colour.

So, the addressable RGB LED ring. I played with addressable RGB LED’s when trying out the Adalight project a couple of years ago – it’s a really fun project to do and I had it running for a while, but took it apart when we changed the configuration of our TV setup. I definitely want to get it going again – possibly another project for the Raspberry Pi at some point. In this instance, the LEDs came in the form of the SparkFun NeoPixel ring – I went for the 60 pixel ring for maximum effect, they come in quarter circles, so you have to solder them together, they’re bright and the accompanying library is really easy to use. Unfortunately, running NeoPixels directly from a Raspberry Pi, whilst possible, isn’t entirely straight forward so I decided to use an Arduino as the controller for the LED’s which meant sussing out serial communication between the Pi and Ardunio via the USB. Again, this is something that I’ve played around with in the past but never really got the hang of until now, and when I did, it made complete sense. The nature of serial communication is that when a message is broadcast, you have a constant stream of bits, which tick through the pipeline, one at a time. Provided that both ends of the pipeline know which bits define the start and end of a message, everything else is just a case of listening for those characters to determine when you should start gathering a message, and when you should stop. This makes it possible to broadcast long messages by concatenating the incoming bits together, beginning and ending the message when the headers and footers are detected. This serial communication allowed me to send the RGB Hex values of the detected colour to the Arduino which would, in turn, set the colour of the RGB LEDs.

As I was developing the functionality for the box mostly off site, I built all of the electronics into a foam board box which could then later be slid into the alcove. This made the installation in the office easy, and just a matter of connecting up power and an Ethernet connection to the Raspberry Pi, for the purposes of remote control over the network via VNC.

The end result works fairly well, although there are a couple of hacks that I used to improve the output a little. In reality, I need to work a bit more on the accuracy of the background subtraction and colour detection. As can be seen above, the masked image often appears quite dark, resulting in a highly saturated colour match, which led me to a bit of a hack. Dissatisfied by the lack of definition in the colour of the lighting, I decided to process each channel (Red, Green and Blue) as normal, but then remove the lowest value, in order to enhance the remaining two values. I know that this is far from ideal and actually causes some colours to be very inaccurate – particularly those which fall at the midway points of the 0-255 scale, but I think that the overall effect is a more defined colour match. For example, if the colour averaging were to derive a value of R: 98, G: 100, B: 105, from a scene, the variance between each colour is extremely narrow, and in this case, every value is of near equal importance in the true representation of the matched colour. Consequently, in it’s current form, the box is good at matching primary colours, but struggles outside of that scale. This is something that I think could be improved with a little more time put into creating a better lighting set up – in terms of both the actual lighting of the box, the brightness settings within the camera module, and a more accurate background subtraction.

The other aspect that I didn’t quite get round to finishing at the time of writing, was the idea of making the box fully automated. My aim with this would be to have the box, operational during specific times of the day, i.e. During office hours, and I would also like to be able to have the box identify a change in scene, in order that it could detect the removal of an old object, and the placement of a new one so that it automatically analyses the new item. In real terms, I think this would be a case of capturing a second reference image, and periodically capturing new images to compare against the second reference image to determine whether a new object was present. I really enjoyed developing this project and aside from its unfamiliar and seemingly open ended syntax, Python was enjoyable to use and offers a lot of possibilities.

]]>https://josephwilkins.wordpress.com/2015/01/17/the-chameleon-box/feed/0lightbox2014_fijosephwilkinsCapture ProcessLightbox StructureThalmic Myo, first impressions.https://josephwilkins.wordpress.com/2014/11/16/thalmic-myo-first-impressions/
https://josephwilkins.wordpress.com/2014/11/16/thalmic-myo-first-impressions/#respondSun, 16 Nov 2014 15:40:00 +0000http://josephwilkins.wordpress.com/?p=497]]>After months of indecision, I finally decided to preorder the Thalmic Myo, it arrived last week and I’ve had a little bit of time to play around with it so thought I’d put together some notes on my impressions so far.

First off, the device itself is fairly well constructed, I think this is worth mentioning because of the fact that Myo’s aren’t actually mass produced in the conventional sense. Thalmic actually manufacture each unit on their own production line in Canada. I really like the idea that the research, development and production line are all more closely connected and, in theory, able to prototype and iterate more quickly and efficiently. I’m not sure what impact this way of working has on the final cost of the device but I think it must have been a brave decision to take, and one which I have a lot of respect for. I remember reading about the challenges faced by the Raspberry Pi Foundation and how, eventually, they were able to bring manufacturing into the UK, again, I think this sort of approach is a good thing which opens up a wider range of opportunities, so hats off to those tech start ups who are nurturing their own talent and keeping things local.

One particular article about the Myo, that caught my attention over the summer was in this interview with Thalmic CEO and co-founder Stephen Lake, on TechCrunch – during which the impression that I get is that one of the main aims with the Myo was to create a device which allows people to dive straight in without the need for any lengthy setup or calibration each time you use it. Although I think that this is an ambitious and idealistic approach, I also think that, in usability terms, it’s a good target to aim for, if you want a broad audience to be able to adopt a new technology, the chances of success are more likely when there are fewer barriers to entry. There’s a lot of cool tech out and about at the moment, but I think the more ground breaking devices; Leap, Myo and Glass in particular, run the risk of alienating broader audiences if they aren’t accessible and easy to hit the ground running with.

In my experience of using the Myo so far, it seems to be fairly robust, it’s quick to start detecting movement and gestures soon after putting the device on, and gyro data; X, Y, Z rotation and acceleration are all fairly reliable. The accuracy of the gesture detection, however, is still a little bit flaky – at the time of writing, there seem to be a fair few false positives, though this appears to be more the case when the Myo is put on and initially misinterprets a gesture. From that point on, it seems that the Myo is unable to reassess a misinterpreted gesture, the only resolution for which, is to remove and re-sync the armband. As such, I’ve found that it’s sometimes necessary to put the armband on, make a few adjustments to ensure good contact and then check through each gesture before you get going. Nonetheless, once you are up and running, it feels pretty reliable.

Upon first finding out about the Myo, there were a few features or use cases which got me really excited, in particular, the idea of having a gesture device which can operated independently of a computer, wires, or the necessity to face a camera or other detection device, other than that which is attached to your arm. The idea of a hands free, lock/unlock gesture based input also appealed and naturally, I thought I’d put a few of the use scenarios, featured in the promo video, to the test. It turns out that even though there is a lock/unlock gesture to effectively start and stop the Myo listening to gestures, attempting to do the washing up, whilst wearing the armband to control music playing in the room, resulted in a few deafening moments when the Myo was unintentionally activated, and the volume turned up : s it’s early days though and I’m sure that the mass of user data sent back to Myo through the Myo Connect app will prove invaluable for the improvement of future gesture detection algorithms.

My first Myo app:

Inspired by the Moff smart wristband, I put together a quick demo app by way of a light sabre simulator, to try out some of the API features. With Processing being my go to prototyping platform, I found a really handy Processing library which seems to provide access to most, if not all, of the functionality that I was interested in playing with.

So far, it’s been fun playing with the Myo, I’m really excited about the potential that it offers, and am looking forward to the improvements in forthcoming API’s. I think it’ll really come into it’s own as a peripheral controller, where it could be used in conjunction with other devices like smartphones, tablets and so on. Much like Google Glass, I think that the Myo would work well to extend the functionality and features of other devices. Imagine, for example, combing Myo with a smartphone app and some WiFi connected bulbs, to allow gesture control of lighting within the home. Directional and gesture data from the Myo could be interpreted by a smartphone app to determine, firstly, which bulb the user is pointing at, relative to the magnetometer in the phone, and secondly, the adjustment that the user wishes to make the that bulb, based on the gesture being performed. The potential for scenarios like this, demonstrate one of the most exciting things about the Myo, – the fact that it is more platform agnostic than other gesture devices like the Leap and Kinect, and this is where I think it gains the advantage.

In summary, the Myo is an exciting bit of kit and I’m really glad that I got one – as ever, I just need to find some spare time to delve a bit deeper and start playing around with some ideas. I think it’s got lots of potential as a wearable – if only the distinction between intended and unintended gestures can be better defined – unfortunately though this is, once again, the classic problem with gesture based input devices – when is a gesture, not an intended gesture?

UPDATE: 20th December 2014

As hoped, Thalmic have been making a lot of great improvements to the API – in particular, it turns out the a lot of people were having problems with the lock/unlock gesture, which they’ve now changed to a double tap between the thumb and forefingers. So far, this seems to be more reliable than the former thumb to pinky lock/unlock gesture.

They’ve also opened up access to the raw muscle data – again, a brave move and I expect that a lot of developers, myself included, will be grateful for the chance to experiment with this.

]]>https://josephwilkins.wordpress.com/2014/11/16/thalmic-myo-first-impressions/feed/0josephwilkinsExperiments with the Leap Motion Controller, one year on.https://josephwilkins.wordpress.com/2014/08/16/hands-on-leap-motion-controller/
https://josephwilkins.wordpress.com/2014/08/16/hands-on-leap-motion-controller/#commentsSat, 16 Aug 2014 14:18:00 +0000http://josephwilkins.wordpress.com/?p=492]]>One morning, in March 2013, I woke with excitement and anticipation, all packed and ready to head out to Serbia for the Resonate Festival. By coincidence, that same morning, I also received the awesome news that I would be getting a Leap Motion Controller Dev Kit. I applied in the summer of the previous year, and hadn’t ever expected to actually get one, so this news, combined with the awesome few days ahead at the Resonate conference had me really excited and looking forward to the next few months of creative potential.

After some correspondence with the Leap Team and a couple of weeks patiently waiting for the device to make it’s way through UK customs, I received the dev kit and started playing. When I initially applied to be part of the developer programme back in June 2013, I cited my interests in possibly developing apps which incorporate gestural interaction and sound which is something I’ve done a bit of in the past, during my MA, and sadly, something which I haven’t done much of since, luckily for me, this turned out to be an area which Leap were interested in pursuing and so a lot of my thoughts about potential projects centered around these themes.

In the 18 months or so since, I have been experimenting with some of the different possibilities that the Leap opens up, of which there are many. This post is a brief overview of some of the demos that I have produced and my thoughts on developing for the device.

In pursuit of the concept that I initially pitched to Leap Motion, I created a few experimental pieces in which I explored different ways to either generate or manipulate existing sounds.

Finger Tenori-on

One of the most obvious and immediate concepts that sprang to mind, was the idea of creating a very simple Tenor ion style sequencer. In this example, there is a play head which constantly moves from left to right on the screen; when the user then places one of more fingers in the path of the play head, a musical note is emitted. The pitch of the musical note is modulated by the height of a finger and the overall speed of playback is affected by the position of the hand on the Z axis of the Leap Motion controller.

Sample mixing interface

Building on the work that I did during my MA in Interaction design, I also liked the idea of experimenting with a sample mixer, of sorts, where the user could have multiple samples represented on screen as small nodes from which the volume, bandpass and sample, could be controlled through gestures. The video below shows a working demo of the kind of interface that could be used with the leap motion. Whilst I didn’t get round to getting the sound modulation controls linked up to this, my main goal was to see how effective a more advance gesture based interface could be used to interact with media.

Interactive Hologram

In addition to purely software based interaction demos, I was curious to see how the leap could be integrated with other hardware. One of the most successful demos that I developed made use of the leap motion controller as an input device for interacting with holographic displays. Shortly after starting work at 9 Yards Creative, I was shown the DreamOC Hologram unit – the DreamOC essentially displays a hologram-like image using the pepper’s ghost effect, whereby a screen is mounted in a box facing downwards onto three two-way mirrors. These mirrors are angled at 45 degrees sloping downwards, away from the middle of the screen, in a pyramid like fashion, the over all effect is that the visual content displayed on the screen appears to float in the air. Working in collaboration with a 3D Modeller, I developed a simple leap controlled product package which the user could both rotate through 360 degrees on both the X and Y axes and zoom in and out by moving their hand over the leap sensor in front of the model.
This proved popular when we demonstrated it at a client demo day as it provided users with a new and interesting way to interact with products in a digital context.

Over the past 18 months, since its release, the leap motion controller has proven itself to be an innovative device, it’s extremely responsive and I think it has real potential. There are a couple of things which have become apparent to me, during the time that I’ve spent tinkering with the Leap.

From the outset, it is clear that this is a revolutionary new input device which, on the most part, requires new thinking in the realms of gestural interaction. The extreme accuracy of the leap also needs to be tamed somewhat to prevent it being over responsive, so applications need to be designed to anticipate certain user behaviours at certain points during interaction, which takes careful consideration and testing to perfect. Often, when working with the Leap, I get a sense that it is far too responsive and doesn’t really compensate for the fact that our hands don’t naturally stay still, even when we aren’t consciously moving them. There is also the fact that the Leap is a device which, in its current form factor, needs to be connected to a computer, limiting the possible applications that it could be applied to. With the current increase in power, functionality and popularity of portable devices, perhaps a Bluetooth version, or even the integration of the Leap within devices themselves would do more to extend the possibilities of gestural input devices.

To begin with, one of my main concerns was that the tracking is extremely responsive, very fast and highly accurate, which is great, as long as you can keep a reference to what you’re tracking. The Leap tends to temporarily and sporadically lose sight of fingers and other points in view. The nature of the tracking means that when a point is lost, either because of lighting, hand or finger overlap, or indeed when a point enters or exits the field of view, a new id is applied and there is no way to maintain the position of that point. This prevents the tracking of individual fingers from being very robust, which is what is needed if more advanced interfaces are to be developed in future. On a positive note, the recent V2 release of the Leap software has dramatically increased the reliability of tracking and it is becoming far more enjoyable to work with. The Leap now seems able to maintain tracking of hands in a much more robust way, and is even able to identify when hands are being turned with palms facing up and away from the controller. It is often easy to lose sight of the fact that the technology is in its infancy and that advancements and improvements are being made constantly and ultimately, kudos is due to companies like Leap Motion and Thalmic Labs for their pioneering work in the field of gestural interaction.

In my experience so far, it would seem that the Leap is a truly great device with a lot of potential, for which there is a lot of untrodden ground in terms of learning how to develop new forms of interaction. It is very much down to the early adopters to determine how simple gestures like selection, drag and drop, directional movements and, so on, should be interpreted. As far as the developer programme is concerned, I’m extremely grateful to have been given the opportunity to work with the leap throughout the developer preview and the support of the leap development team has been second to none. It’s been a great opportunity and I’m excited to be working with some great new tech, which I hope will broaden the discipline of interaction design.

]]>https://josephwilkins.wordpress.com/2014/08/16/hands-on-leap-motion-controller/feed/3leap2014_fijosephwilkinsInteractive Gesture Wallhttps://josephwilkins.wordpress.com/2014/06/21/464/
https://josephwilkins.wordpress.com/2014/06/21/464/#respondSat, 21 Jun 2014 13:49:00 +0000http://josephwilkins.wordpress.com/?p=464]]>I spent the earlier part of this month finishing up a large interactive gesture wall which was exhibited at the Eular medical congress in Paris.

For the main attraction on the exhibition stand, I developed a gestural user interface which allowed visitors to guide molecules around a large video wall, measuring approximately 6 metres wide by 4 metres high, with the objective of joining them to other molecules. Uniting molecules would cause them to bind and expand to reveal a small pieces of information in the form of infographics.

The application for the wall was developed using the Processing framework, which is easily one of my favourite tools when it comes to larger scale interactive projects as it’s generally really solid, capable of handling rich visual content, and has loads of amazing libraries and hardware integration options. It’s a great framework with a really creative, inspirational and helpful community of developers backing it. I’ve donated to the foundation and would definitely urge anyone else using it, to do so too.

The hardware for the wall itself was comprised of 24 bezel-less screens, stacked 6×4, with each column of 4 running off of a PC, 6 of them in total. All 6 PCs were connected to a server which handled the synchronisation of content moving across the whole wall. Connected to each PC was a Microsoft Kinect which handled the user input by providing X and Y coordinates of the user’s hand. The interaction was very simple and allowed the user to guide molecules around, the molecules also had attraction and repulsion behaviours, allowing them to interact with other molecules, by snapping towards target molecules and avoiding other molecules to make space when information was revealed.

As with all of the best projects, there were a couple of really significant challenges that were faced in the development of the gesture wall, so I thought I’d cover a few that we came across, the solutions, and perhaps how we’d do things differently next time.

Synchronisation:

From the off, one of the first things we had to figure out was how we would display and run interactive content over such a large display area. Since we were using the Processing Framework, I quickly found out about Daniel Shiffman’s ‘Most Pixels Ever’ library. This excellent library allowed for content to be synchronised across separate applications so that we could create the impression that a swarm of molecules was drifting over the entire display area. Most Pixels Ever utilises a Java based server which each client in the overall display area connects to.

Once up and running, the server provides each client with information about the positions of on screen graphics and the current frame count. The clients then update accordingly and then return their positional info back to the server.

Most Pixels Ever allows for an amazing degree of felxibility, including the ability to define very specific screen layouts, even those in which screens a spaced apart, or don’t conform to a grid-like pattern. It also allows for settings like the frame rate to be adjusted over all clients, and provides really comprehensive debug output which helps when trying to set everything up. Can’t recommend it enough!

Gesture:

As mentioned, we used a series of Microsoft Kinects to provide the input for the gestural interface on the wall. I did a fair bit of experimentation with the Kinect in order to determine the best way to position it, and how best to translate the user’s arm movements into meaningful gestural input. During this experimentation process, we were considering ways to maximise the use of the Kinect by attempting to capture both, the area immediately in front of the wall to pick up a user’s gestures directly with the content on the wall, as well as the area behind the directly interacting users to catch people passing by the wall.

With this idea in mind, we initially we considered mounting the Kinects above the wall, looking down over the user from the top of the wall, angled at approximately 45 degrees so as to position the Kinect’s field of view to achieve the coverage described above. The only issue with this positioning of the Kinect is that the Y and Z axis are projected at angles which don’t run parallel with the real world, which makes them harder to utilise. The Y axis ascends in such a way as to be constantly rising up and away from the base of the wall and the Z axis extrudes downwards and away from the wall.

In the interest of keeping the development of the project going at a swift pace, we decided to opt for a different solution where we positioned the Kinect at the foot of the wall, situated behind a mirror, angled at 45 degrees, such that the Kinect was able to see both over the top of the mirror, to catch the background interaction, and upwards, where the bottom half of the Kinects’s view is bounced directly up in the air which, being situated at the foot of the wall, allows the bottom half of the Kinect’s view to capture gestures being performed directly in front of the wall. This solution worked fairly well, save for a few issues with lighting and a lot of tweaking to prevent the Kinect’s effective field of view from throwing too short – something which we realised is something that it tends to do, if the view is obstructed.

This final solution allowed us to capture user input directly in front of the wall and some background tracking of passers by, which didn’t work so well due to the blackouts in tracking where users were standing immediately within the field of view. I think this is a use-case where an overhead tracking method would have worked far better.

Whilst, unfortunately, we didn’t have time to pursue the overhead approach on this particular project, I’ve been considering ways that we could implement it. Ultimately, the X, Y and Z positions, observed within the view of the Kinect, angled 45 degrees down from above, would need to be translated in some way to provide a real world coordinate. I think this may require the use of the Polar coordinate system as opposed to the Cartesian coordinate system, in some way and although I haven’t completely sussed it out yet, I’m sure I’ll figure it out at some point in future!

I think that one of the biggest issues that we battled with was that it is essential that you are able to control the environment, as far as possible. As I mentioned, we experience some really strange problems with the Kinect having it’s range cut really short because of the fact that it’s view was obstructed too early on – it’s almost as if there is some form of adaptive ranging at play where the effectiveness is limited by how far the Kinect can see. We also found that lighting played a significant role in affecting how well the Kinect could see things, we often had dark patches where the Kinect appeared to be locking on to light sources in it’s view. Apparently though, with the various modifications and updates to the hardware, this won’t be so much of an issue with the next generation of Kinect, which will come as a huge relief.

On the whole, the large gesture wall was a great success and there was a lot to be learned, purely through the observation of visitors and how they interacted with it. Although devices like the Kinect, PlayStation Move, the Wiimote, and more recently, the Leap Motion, have been emerging over the past few years, I would estimate that gestural interfaces of this kind, i.e. ‘the failing of limbs in the air, in order to manipulate some form of digital application’ is still a relatively new and unfamiliar paradigm within the day to day experiences of an average user. I say this with caution as, of course, people are indeed using these interfaces, I think the real challenge, however, lies within the fact that a common gestural language is still in it’s infancy. Whereas, we can easily associate with the practice of using a computer mouse, where using left click allows us to select something and right click often allows us to explore the further options available from a particular item on screen in the form of a context menu, the same sort of conventions don’t yet exist for gestural interfaces. What this means is that, at this stage, we often have to adopt a straight forward and logical approach to gestural interaction. This also means that it’s a really exciting time to be working in this field as people developing these interfaces have a real opportunity to come up with novel and exciting ways to bring gestural interaction into a more public domain, through the development of usable interactions and feedback.

]]>https://josephwilkins.wordpress.com/2014/06/21/464/feed/0josephwilkinsInteractive Gesture WallThe wall, in developmentGesturingPhotoboothhttps://josephwilkins.wordpress.com/2014/04/21/photobooth/
https://josephwilkins.wordpress.com/2014/04/21/photobooth/#respondMon, 21 Apr 2014 11:56:13 +0000http://josephwilkins.wordpress.com/?p=439]]>A few months back, we went to a friend’s wedding at which they had one of those photo booths where you can bundle in with a load of friends, take a picture and take away a printed photo. Until I saw one up close, it never really occurred to me just how straight forward they actually are, and how easy it would be to put one together. With my sister’s wedding coming up, I thought I’d have a go at making my own.

In principle, a photo booth of this nature consists of a camera, a preview screen, a printer, and some sort of trigger, be it a touch screen or some other input device. I had most of these bits and pieces lying around in some form or another, apart from a suitable printer, for which I used a Canon Selphy CP 530, which I managed to pick up form eBay for about £11.

In terms of the software, I created an app using the Processing framework, in conjunction with some basic Automator apps and folder actions. For the functionality of the photo booth, I managed to break the functions down into 4 main areas which complete the process of capturing a photo.

Firstly people need to be able to see a preview of what the camera can see, in order that they can pose within the shot. For this project, I used a Canon 350D DSLR, ordinarily, I would have like to have used the movie mode featured on most SLR’s, to provide the video preview for the photo booth, however the 350D doesn’t have a movie mode as it’s a relatively old model. I messed around a fair bit with different pieces of software including the bundle Canon software, and toyed with the idea of using the Canon SLR SDK, but actually found that the easiest way to create the preview, given the time and resources that I had, was to hook up a webcam which would display the area covered by the camera. I then mounted the webcam above the DSLR on a tripod so that it would roughly show what the DSLR sees. It’s a bit of a long way round but seemed to be the best solution in this instance.

The next main piece of functionality was providing the user with the ability to release the shutter. To do this, I made use of a USB button which behaves like a keyboard with a single key. With this, I was able to allow Processing to preform a function on detection of a specific keycode that was fired by the USB button. When the Processing app detected this key press generated by the USB button, I had it trigger a countdown, of approximately 10 seconds, to the shutter release allowing those using the photo booth time to get ready for the photo. The countdown was displayed on screen to indicate the time remaining.

For the shutter release itself, I made use of the ‘open()’ function within Processing, which allows you to open an application or file with the native launcher of the computer on which the app is running. The ‘open()’ function loaded a simple external application which I created using Automator. The single function of this Automator application was to release the shutter on a connected camera and save the captured image to a designated folder. Automator apps are extremely easy to setup and provide you with easy to access to many system functions in a secure and hassle free way. I found it to be the perfect solution for the task in hand, especially considering that the alternative was to wrangle with Canon’s own software or SDK!

Once the DSLR captured the photo, I then set the Processing app to check the capture folder for the most recently generated file. When found, the file was loaded onto the screen, and displayed for approximately 15 seconds, to allow the user a chance to both review the final image and decide whether they would like to print the image or not.

If the user clicked the USB button again during this review period, a second Automator app was then used to copy the most recent file in the capture folder (the photo currently being reviewed) to the print folder. A folder action on the print folder, again created using Automator, was set to print any files which were placed within it. Once the review timer had elapsed the whole would process reset, allowing the next photo to be taken. Throughout the entire process, messaging and feedback was provided on screen to prompt the user of their options throughout.

As mentioned, all of this functionality was wrapped up into a Processing application, which made it easy to launch full screen and coordinate all of the actions required to go from pose to print.

The use of Processing, in conjunction with Automator, made the development of this app fairly straight forward. The bulk of the code within the Processing app was there to deal with things like displaying the preview image, triggering the shutter release, reviewing the photo and then printing the photo if required. I also included a few settings to adjust things like the orientation and mirroring of the preview image, just incase we needed to set the webcam up in a different position, and the timer functions which dealt with the scheduling different functions and events within the app.

As a simple photo booth project this was a lot of fun to develop, and the end result proved to be quite popular. I also think it would be fairly easy to enhance and develop this project into something a bit more substantial in future. There are some really clever examples of photo booth set-ups out there, from simple shoot and print booths like this example, to more inventive and entertaining solutions like those which allow users to record short video clips on high frame rate video cameras, which can then be played back in slo-mo, right through to the crazy Purikura photo booths which include all sorts of image manipulation options and post production features. I think it’s a really interesting area of recreational tech, and definitely something worth keeping an eye on…

]]>https://josephwilkins.wordpress.com/2014/04/21/photobooth/feed/0photobooth2014_fijosephwilkinshardwarePhotoboothiBeaconshttps://josephwilkins.wordpress.com/2014/03/11/ibeacons/
https://josephwilkins.wordpress.com/2014/03/11/ibeacons/#respondTue, 11 Mar 2014 16:29:11 +0000http://josephwilkins.wordpress.com/?p=437]]>iBeacons are small coin sized devices, which emit a signal in the form of a radio frequency. They are based on the bluetooth low energy specification and many mobile devices already on the market contain the technology required to pick up the broadcasted signal. Essentially, they can be used to trigger an event or notification on a user’s device, when a user has a compatible application open. In practical terms, they are being marketed as an alternative to Near Field Communication (NFC) technology and are capable of being put to similar uses as NFC technology, though with a greater range.

I’ve been experimenting with iBeacons over the past month or so at work, trying to figure out the possibilities of what can be done with them. Experiences have been varied so far, with software based beacons, mostly in the form of the Estimote App (Android/iOS), proving to be really effective, and hardware based beacons being very erratic and tricky to work with.

So far my experience is with detection via an iOS app, built using the Titanium platform, in conjunction with an excellent iBeacon module. The module basically allows you to detect iBeacons within range and then handle the incoming data as you wish. The incoming data from each iBeacon includes things like the beacon id, rssi, broadcast strength and accuracy properties, along with a few other pieces of data which can be used to determine proximity and the iBeacon’s identity.

One of the biggest challenges so far, as I’ve mentioned, was in getting hardware iBeacons to work in a meaningful way. With the setup that I have been using, an event listener picks up the broadcasted signal, which is transmitted approximately once per second. The first problem is that, for some reason, the beacons transmit an extremely erratic signal, not only in the form of a signal which fluctuates at around +/- 10, it also throws out some anomalous 0’s so, the raw signal form each beacon is far from steady.

Tackling the anomalies was straight forward, simply filter out the 0’s from the incoming data, after that, you’re left with a more smoothly undulating signal. In order to achieve a consistent distance value, we can borrow similar sorts of techniques which are employed in other scenarios where it’s necessary to reduce spikes in an incoming data stream, for example, within audio processing, or stock market analysis. A technique known as Simple Moving Average (SMA) allows us to achieve a smoother stream of data by storing the past n# values within an array from which we can calculate an average value, whilst removing old values from the array as we add new ones, in order to maintain a small and timely, but more accurate sample. Using this array, we calculate the average.

This technique seems to be working really well and allows features like proximity sensing to work a lot more effectively. Prior to the SMA algorithm, beacons would jump in and out of view constantly, which made it really hard to stay fixed on one beacon, or even stay focused on that which was nearest.

Other than that, I’m still working on methods to improve the reliability of iBeacon signals and hopefully we’ll be able to make use of the technology in a meaningful way… as is often the case with the early adoption of most technologies, the initial learning curve can slow things down a bit, but as the nuances of the technology get worked out, it’ll be interesting to see how iBeacons get used in the real world.

Here are a few articles/posts which I found really useful in getting started with iBeacon technology:

Earlier this month, I flew out to Düsseldorf, for the Beyond Tellerand: Play & Make conference. It was the second of two conferences that I attended this year, both of which were outside of the UK, and it was great to gain some new perspectives on the world of creativity in software and hardware, beyond my usual trip to Flash on the Beach/Reasons to be Creative in Brighton.

The Play & Make conference featured talks on a broad range of topics including UX/UI, application development, and then emerging technologies in areas like physical computing, including the use of computer vision and platforms like Arduino and Raspberry Pi. The Capitol theatre was a great venue with a really nice auditorium layout consisting of small lamp lit tables in the front seating area. Following, are some of the observations and notes that I made from the speakers over the course of my two days at the conference.

Day one:

Carlos Ulloa

Carlos is someone I’ve seen speak a few times now and it is always apparent that the projects produced by HelloEnjoy, the interactive studio which he is a part of, are the sum total of some outstanding technical achievements. In this particular talk, Carlos spoke about HelloEnjoy’s latest project, ‘HelloRun‘, a game built using WebGL with the HTML5 stack. The game itself is simple enough, you move, in first person through a tunnel in which there are barriers which you have to dodge to avoid. The real technical achievements lie in the details, and examples of this can be seen in aspects such as the lighting of the scenery, and the way in which the audio API is used to make the lighting respond to the beat of the music. Carlos provided a really in depth look at some of the challenges that were overcome within the project.
One of the key factors that I found most impressive was the fact that it is also possible to play the game on mobile devices, provided that WebGL is present, leading ‘HelloRun’ to be a truly worthy example of a web app

Jared Ficklin

I first saw Jared Ficklin speak at Reasons to be Creative in 2011, during which he demonstrated and spoke about some of his experimental work with sound, fire, smoke and the Kinect sensor. I also remember him talking about the interactive opening party for South by Southwest in 2010 which involved the development of some really interesting hardware implementations including projection mapped portaloos and networked RFID check in points.

Within his Play & Make talk, there were some really interesting considerations about the present state of interaction design and how we can expect this to develop in the near future. He spoke about some really interesting themes but my main take away was his emphasis on the need for more heads-up, collaborative computing experiences.

Jared emphasised that much of the technology we use on a day to day basis claims to enable us to be more social and connected, yet it requires us to be physically hunched over a relatively small screen. This is also often an experience suitable for individual users and effectively isolates us from the rest of the world. In the interest of making technology more inclusive and cooperative, rather than heads down – there is a need for research and development into technologies and computing interfaces which are integrated at a room level, keeping us ‘heads up’ and collaborative, rather than technology being condensed into the mobile device form factor which keeps our heads down and effectively shutting out the world around us. This makes a lot of sense to me and I think there is a lot to be said for how current technological trends will affect social behaviour in the long term. It’s also the reason that I question the worth of upcoming devices like the Oculus Rift, beyond gaming.

Other great speakers on day one included Florian Schmitt, whom I had seen at Resonate earlier in the year showing his work with both Hi-Res and Nanika, and Mario Klingemann, who, in the last few years has branched out into some really interesting avenues of development. I particularly enjoyed hearing about his work with FAT Lab Berlin – especially so, the working typography gears, I think this was a great little project and there could be more in this concept and the idea of creating mechanical story books of some sort. One of the really poignant ideas that came across in Mario’s talk, for me, was that stochasticity can often lead to more original ideas, rather than constantly referring to the usual sources, and things that have already been done. I also found James Patterson’s talk really enjoyable. In particular, he showed some of the work he had been developing in collaboration with another interaction designer, Amit Pitura (http://www.pitaru.com) which I found really impressive. Two really outstanding examples included: http://www.rhondaforever.com and http://sws.cc/sws2003.html. He also taught us how to fold the ‘Boolean’ paper plane design, because analogue is still fun too.

Day 2

Day two began with a demonstration of the new Microsoft Kinect V2 sensor by Gunter Logemann. He demonstrated some of the new key features including the vastly improved tracking resolution, thanks to an HD camera, which now makes it possible to track up to 6 people, each with 25 points of joint information. Improvements have also been made possible through the addition of an active IR sensor which allows visible light sources in the room to be filtered out, thus reducing interference from lamps and other lights being used in a room. Of the features that we were shown, the new Kinect appears to be a huge improvement on the last, offering far more robust tracking.

It also appears that Microsoft have taken a lot of user feedback on board and are attempting to implement a more natural user interface. Where the initial Kinect relied a lot on the use of devices like time activated switches, the new Natural User Interface, or NUI, uses more familiar gestures like taps and hovers and is capable of recognising hands as hands, as opposed to points, hopefully allowing for more natural gestures to be used. From Gunter’s talk, it certainly seems apparent that, following the initial release of the Kinect, Microsoft are working hard to develop the Kinect as an input device.

Eva-Lotta Lamm:

Eva-Lotta Lamm’s talk took the form of a sketch workshop which I really enjoyed. Lately, I’ve been trying to get more into drawing and sketching, where appropriate, as it’s often a far more effective way of putting a point across and I think that lends itself far better to both the conveyance of a message as well as during the process of generating ideas.
The very act of taking notes during the process of brain storming, for example, often causes us to short hand or abstract the essential meaning from an idea, in order that we can get that point down quickly and move on to the next point. This process may detract from the idea itself, and if we must go through the process of making marks on paper, in order to create recognisable symbols for future recall, why not sketch a scenario, instead of describing it?
As part of the session, we had a quick introductory lesson which covered the basics of sketching which I can see being useful in the process of getting ideas on to paper when prototyping new concepts. Eva highlighted a website that she and a few collaborators have developed, which provides inspiration for practice doodles: http://evalotta.pancakeapps.com
I’ve seen Eva featured in the bill for a few previous conferences and whilst I have seen some of her work, regrettably, I haven’t had the chance to see her speak so this was a great opportunity and I enjoyed the talk a lot.

Niklas Roy:

I’ve been following Niklas Roy’s work ever since I saw his “My little piece of privacy” project. In this piece, he has combined motion tracking with a motorised net curtain in an attempt to prevent people peering into his workshop from outside. He’s a developed a really humorous solution to the problem which provokes playful reactions. I also really like the way that the reaction times of the curtain movement aren’t quite quick enough to track passers by, and that sometimes anomalous results in the tracking cause the curtain to twitch and stutter. We often expect more than we should from technology and the random utterances made by the Curtain Control device, almost give it personality.

Niklas Roy’s other projects, many of which stem from the field of physical computing, explore interactions and ways in which mechanics and computing can be combined to provide interesting, and often very creative, solutions to problems.

I really enjoyed Niklas’s hearing about projects – they all seems to execute an idea in a really playful and creative way. I like the characteristics and charm that his machines and projects often have, it makes them less machine like a bit more human.

Beyond Tellerand: Play & Make was a really enjoyable conference with some really great speakers, I wrote tons of notes and genuinely found it really hard to distill everything that inspired me into a single blog post so I’ve hopefully capture the key highlights for me personally. I really hope that this sort of event runs as part of the Beyond Tellerand series again as I think it’s more broad and indicative of some of the paths that interaction developers are taking, in particular, but not limited to, those who are developing their skills, post Flash.

]]>https://josephwilkins.wordpress.com/2013/12/21/beyond-tellerand-play-make/feed/0btpmconf2013_fijosephwilkins20131228-003754.jpg20131228-003922.jpg20131228-004138.jpg20131228-004301.jpg20131228-004506.jpg20131228-004627.jpg20131228-004852.jpgGTA Style Hidden Packages “IRL”https://josephwilkins.wordpress.com/2013/12/12/gta-style-hidden-packages-irl/
https://josephwilkins.wordpress.com/2013/12/12/gta-style-hidden-packages-irl/#respondThu, 12 Dec 2013 04:28:56 +0000http://josephwilkins.wordpress.com/?p=361]]>So yesterday my flight home, after Beyond Tellerand: Play & Make*, was cancelled due to fog in London. Consequently, I got stuck at Düsseldorf Airport for about 12 hours, waiting for the next available flight. In my boredom, I decided to do what GTA has taught me to do best…hunt for hidden packages…true to form though – I ran out of time whilst trying to find the last one! 8 )
Any sightings of present No:2?