Testing And Related Discussions In Software

Crunch time. Day 2 comes and so does the Exploring App (In)Security workshop alongside one of my most important testing mentors, Bill Matthews.

We had been planning this workshop for some time, and we really wanted to make this work for the attending delegates. Bill had pulled out all the stops to create a really brilliant learning resource in the Ace Encounters web application, and together we planned the learning objectives we wanted to achieve.

Our aim was to provide a safe learning environment where the delegates could learn about security test design techniques, the key vulnerabilities in web applications and how to exploit them. It was also our intention to elicit discussion around these issues in the context of software testing, rather than hacking.

Bill Matthews in Action!

There were lots of great opportunities for Bill and I to learn as well, feeding off the needs of the attendees, and also their experiences. It’s the best way for us to get better at presenting the content, making it more relevant and exciting for everyone. Here are some photos of the day, where we got to work with some really great testers!Let’s Test is famous for it’s more social activities. You can’t go far from the conference venue, as it is in the middle of nowhere. So, we all have to create our own entertainment.

As Day 2 drew to a close and after a great chat with some awesome people in The Test Lab, a few of us retired to the games room – ostensibly to play pool, but as always things descended into testing games and chat!

This is part of the attraction of Let’s Test, where you can just hang out, with a few beers (or whisky in our case) and talk about test, the universe and everything.

Chris Chant, Dan Ashby and Phil Quinn

On to Day 3, which was again a fantastic day of learning. This conference was my first chance to speak to many testers that I had admired and followed for sometime – such as Patrick Prill – @testpappy on Twitter. I hooked up with Patrick, Christina Ohanian and Dan Ashby at lunch time, and we did an impromptu recording of Testing in the Pub! I can’t wait for that episode to come out.

Patrick Prill

The morning lead me to more facilitation responsibilities, this time trying to manage the events at Jean-Paul Varwijk’s very well researched presentation and debate on the proposed ISO 29119 standard.

It wasn’t my job to get involved so much in the debate, but ensure that all the participants of the meeting at least got a chance to take part (If they wanted to) and ensure there was some sort of order to the questions, follow ups and burning issues being raised.

There was a lot of passion in the discussion. Clearly this issue has sparked much interest and concern within the context driven testing community. My main issue however that there was no real moderate or conflicting view arising from this discussion – most if not all people who spoke up had little that was positive to say about the proposed standard, or opposed it out right.

Still, Jean-Paul had presented a tonne of material he had researched and gathered over time, and presented a cogent argument in as balanced a way as he possibly could. All in all, I am glad I volunteered for this session, as it allowed me to see testers debating in action!

Jean-Paul Varwijk

Without doubt the highlight of Day 3 for me though was the fantastic session “Coders to the Left” lead Jan Eumann and Philip Quinn. This workshop encouraged us to work in pairs and small groups, with each activity with a different focus, for example working as a tester, developer or observer.

They had created an excellent resource for learning via a GitHub project called Fixture Finder. It essentially allowed you to search football match fixtures, using date and country as search criteria. More than that though, the workshop allowed us to explore what working like a developer might be like – and it was a challenge.

Rather than just finding bugs, we would isolate the cause and fix it on the fly, within our own instance of the app in Chrome. There were some very interesting bugs to find, such as blatant security flaws, or little bits of code that stripped search results from the list, or tampered with the results of football matches under certain conditions.

I know a bit of code. Not so much that it would allow me to call myself any kind of developer. I can use code, and other tools to help me solve testing problems. However this activity really did let us get to grips with how testers and developers can really work well together, reducing and improving the feedback loop as we test and code together. A brilliant exercise in collaborative learning.

Jan Eumann and Phil Quin

Anders, Dan and me pairing up

So, as my first experience of Let’s Test draws to a close I want to reflect on what has been a most rewarding and exhausting experience in equal measure. The learning from the workshop I ran helped us feed this learning into the following session at Nordic Testing Days, yet it made me realise that I don’t really blog much about security. I should rectify that.

Let’s Test allowed me to engage deeply with my personal approaches to testing, and what I value about myself as a human being. The impromptu chats, podcast recordings, Reiki healing workshops with Dawn Haynes, the testing games, workshops and talks I attended all helped with that. I do attend to go again, as it is such an intense and engaging place to be.

It is now almost a week since I arrived at Let’s Test near Stockholm in Sweden. I had heard a lot about Let’s Test, not least from my Weekend Testing colleagues Amy Phillips and Neil Studd. It was there this time last year that they decided to restart the Europe chapter. I had also heard a lot of good things about the conference from others in the community, all of which were overwhelmingly positive. So, as I recall my feelings and trepidations about attending and working at Let’s Test, I do it now with a renewed vigour regarding my career and learning.

The venue, nestled in a Swedish rural idyll on the Baltic coast close to Stockholm, is the perfect place. To say that it is beautiful is an understatement. The conference centre has the perfect combination of location and facilities that create a fantastic environment for learning, and of course, the socialising! After all, the conference is organised for testers, by testers.

In addition to this challenge, I was not only running a workshop on security testing with Bill Matthews (more on that later) but I had also volunteered to be a facilitator. This meant that the workshops or talks I had volunteered for, I had to assist the speaker as much as possible with setting up and equipment, generally being a gopher for them. During the “Open Season” portion of the sessions, facilitators had to manage all the questions fielded by the attendees. The conference organisers had given us all K-Cards, to allow us all to take part fairly in the discussions. If you want to know more about K-Cards, check out this blog by Paul Holland – The history of K-Cards

Ben Simo – “there was not a breach, there was a blog’

Day One

The opening keynote was in a word, fantastic!

Ben Simo is a tester that I have been following for some time. His experiences and learning from attempting to organise health insurance on for his family would have been hilarious, if it hadn’t been so serious. “there was not a breach, there was a blog” was a fascinating journey through the issues and problems surrounding the release of healthcare.gov, the US Government website and initiative more popularly known as Obamacare.

Not only were there many functional, usability and performance issues with this site upon release, but also a huge range of potential security vulnerabilities. At the time, Ben blogged about these issues, trying to make the government aware of the problems and ultimately found himself somewhat reluctantly being the subject of media interest.

Ben is an eloquent and humorous speaker, who is extremely skilled and knowledgeable about his craft. His experiences also reflect strongly upon my recently learning in the sphere of security testing and as a result, the most significant takeaway I had from this talk was the matter of ethics when reporting issues in live, public systems. Ben emphasises the need to constantly be aware of the ethics of testing, and not harming the site. All in all, a brilliant start to proceedings.

Next up was an exciting and challenging workshop run by Emma Armstrong – “Equipping you for the unexpected challenges of testing”. I’ve known Emma for a while, but I’ve never seen her speak or run a workshop.

Emma Armstrong -“Equipping You For the Unexpected Challenges of Testing”

Emma had created a huge range of resources and a challenging application for us to investigate. Emma’s workshop encouraged us to examine and use a wide range and techniques and thinking in order to solve a testing problem. I really love pairing and working in groups with others, so this workshop really resonated with me. There is no better way to learn than to learn from others, in practical situations.

One of the best takeaways I had from this whole conference was during this session. I was pairing with two other testers, one from Sweden, the other from Romania. We discovered that our cultural differences, and in turn our similarities, often drive our thinking while testing. It’s not often I get to pair with testers from outside the UK, so this was a fantastic experience.

Our backgrounds and values often will impact the way we think about testing, and the problems we uncover – for example – a “Title” field would be almost unthinkable outside the UK, yet in the UK to not to be able to select whether you were Mr, Ms, Mrs, Miss or even a Captain or Lord would be equally strange.

After lunch I attended a half day workshop run by John Stevenson – “A Journey towards self learning”. I was facilitating this session, so helping out John with logistics and cold beverages! Despite my responsibilities preventing me from taking many notes, this workshop was and extremely engaging exploration of our own learning.

John Stevenson – A Journey towards self learning

One of the major themes of the workshop was how constraints on information gathering can impact the quality of our learning and analysis of the information we gather. It can inform our opinions and how we apply values or biases to the learning we do.

One great example of this was a particular exercise. The group had to divide into three where each team had a particular task – discover as much as they could about the conference venue, with particular focus on the local flora. However each team had a major constraint imposed upon them – one was only able to use internet resources, another group could only use observations of the local environment, and the third could only speak to people at the conference venue. I went around with the third team to make sure the rules were adhered to.

The results were impressive and eye opening – whilst the team who had access to the web were able to gather a lot of data very quickly, they didn’t have the richness of data gathered by the other teams. It wasn’t easy for the other teams either, where it was fairly hard for team three to use information other than that gathered through word of mouth, as there was so much visual data to gather. Also, we were able to observe discrepancies and contradictions in the information that had been gathered. Its up to us as testers to be able to be mindful of our values and biases when analysing data, manage and work within constraints. John’s workshop was a fantastic way to engage with our own learning in an active and positive way!

All in all a fantastic start to an intense few days of learning! I’ll be blogging about day one and two over the next few days. Watch this space!

It’s taken a while for me to digest and understand the impact of attending MEWT a couple of weeks ago now. I normally try and blog quickly after an event, whilst my memory, notes and personal response are fresh. In this case, I haven’t been able to do so.

Visiting a conference or attending a few track talks and workshops is an exciting experience. There is always an opportunity to learn more about a technical skill, tools and current thinking around testing. Never before have I been able to learn very much about myself as a tester, and as a human being than I did at MEWT.

Set in the fabulous surroundings of the Attenborough nature reserve in Nottingham, MEWT (Midlands Exploratory Workshop in Testing) is a very intimate workshop day hosted by Richard Bradshaw, Vernon Richards, Bill Matthews and Simon Knight. I felt extremely privileged to be invited to attend, so I wanted to ensure that the content I provided was both pertinent to the topic and expressed my personal challenges with communication, some of which I will talk about here.

The Attenborough nature reserve, Nottingham

My talk was Communication, Influence and the Geek, the slides for which are available from the MEWT website.

During my time on this planet, and latterly as a software tester, I have encountered a few challenges to communication. Being a geek, which to some is a pejorative term for someone who has a deep interest in science, technology, certain hobbies or non mainstream culture; can present certain problems for folk who identified as such, or who have been labelled as such by others.

The Dalek Supreme in “The Stolen Earth/Journeys End” in BBC TV’s Doctor Who

Communication is an exchange of ideas and viewpoints, as much as it is about information and facts. Its about disecting and evaluating the information that is presented to you in the context of the emotional feedback you have to it. Testing, in my view, is partly an expression of that.

In deep debate at MEWT

I won’t dwell too much on my personal experiences here, because they are not for this place. However, the feedback from the peers that I met and worked with at MEWT was greatly positive, and nourishing. It has fed my desire to learn more about my craft, and support others who wish to learn more. Whilst we should be mindful not to label ourselves, allow ourselves to get pigeon holed by how either society, others and even our own prejudices, it is important to recognise and play to your own strengths.

Simon, Vernon, Christian and Christopher

The environment created at MEWT allows professional, non judgemental, challenging but friendly debate around the ideas and thinking generated during the day. Ahead of this session I was terribly nervous about sharing some of my deepest thoughts and feelings on the problems I have faced as a tester. I am not sure I could have put all this out in the open in any other conference or workshop.

Dorothy Graham

Raji Bhamidipati

This was a message that has been impressed upon me not only by the MEWT attendees, but also a number of my colleagues, to whom I will always be grateful. One point was made to me, and that was to not be afraid to embrace the influence that my personal interests and idiosyncracies have upon my approach to testing. They make me who I am, and it is that allows me to add value to my employer and those around me.

I make no bones about it, I love this conference. No other expression of emotion comes close. Its almost up there with my wife, family, friends, my cat and Doctor Who. (And to anyone that knows me, that is a pretty big deal)

Regardless of the quality of the conference track, speakers and workshops, this annual event is now rapidly becoming a part of me, my learning as a tester and driving my desire to evolve my testing. It also helps me support and mentor other testers – both those I work with, and those I don’t.

As I mentioned in my previous post, where I previewed TestBash 2015, if it hadn’t been for TestBash I most likely wouldn’t be working where I do today, with a company I enjoy working for, and a team that I admire and value. I also wouldn’t have had the courage to do any public speaking or workshops if I hadn’t attended TestBash in 2013. As long as it is running, and as long as I can attend, I will go. With some luck and preparation, I hope to be more involved in TestBash 2016!

Now with the context of this blog post set out, I’ll try to present my ‘impartial’ review of this conference. It’ll be hard!

For the last three years I have made a pilgrimage back to my home town of Brighton to attend TestBash. Each year it has produced a different mix of learning, excitement, comradeship and an emotional exhaustion that my friend and BrighTest organiser Kim Knup has aptly described as the post TestBash blues. Through TestBash, social media acquaintances have become colleagues in testing, and in some cases firm friends. I may only see them for a few hours a year, but for that, above anything else I am grateful to Rosie Sherry, Simon Knight and all of the Ministry of Testing team that run the event.

Brighton Pavillion at night

I took the photo above of Brighton Pavillion, whilst having a fantastic chat with Stephen Janaway on our way to the meetup on the Thursday night. And it is this that indicates the value to me of TestBash as a whole. It’s all about the conversations. Stephen was not the first great chat that weekend, nor was it the last. We discussed testing, my poor recollection of the geography of Brighton seafront, our upcoming conference talks and workshops and even family. I suppose you could say that the testing community, formed around this conference has become as sort of family to me.

Here we are at dinner with Chris Chant, Vernon Richards and Rosie. For me, the conversations start with the small events and gestures like this, and reminds me that I owe Rosie dinner! It had become a bit of an in joke that Vernon was going to wear a tutu on stage on the conference day, and in the end he did, but not in the way you might expect. More on that later. I was lucky enough to hangout with some of the conference speakers and workshop facilitators at dinner, discussing their experiences and feedback on the day. As conferences and workshops go, it very good value for money, as the Ministry of Testing is able to attract some high calibre speakers and contributors every year from across the community, even just to attend!

Chris, Vernon and Rosie at dinner

Sadly, I was unable to attend the workshop day on the Thursday. However, I was able to catch up with some folks at the end of the day down at the Brighton Dome. There was an open meetup and test gaming session to wrap things up, so I watched a round of Set, and led a few testers in a few rounds of Zendo. If it hadn’t been for a lunchtime learning session with my colleague and friend Chris Simms a few months ago, I wouldn’t have had a set of rules in my head ready to play! All power to the test community. Even though he hadn’t attended this year, Chris’s impact was felt from afar!

Ryan and Danny at the meetup

So, off to the meetup, at a bar I hadn’t been too since my early 20’s. We took a minor detour on the way, but got there in the end. Here is my colleague and good mate Danny Dainton, enjoying a drink with Ryan Rapaport, a representative of one of the conference sponsors QA Symphony. (Shameless Plug 1: I use their tool QSnap, it’s pretty good).

The greatest value of TestBash for me comes from the conversations had at meetups like this. Sure, there was a lot of talk about testing, about our experiences of testing, our learning from various books and speakers, the relative merits of one conference over another, the relative merits of one beer over another. Here I was able to catch up with my (Shameless Plug 2) Weekend Testing Europe colleagues Neil Studd and Amy Philips, and plan our ground breaking trio 99 second talk for the following day! I also managed to grab conversations with; Matt Archer, about the Ministry of Testing Dojo and Abbie Maddison, the new runner of the NottsTest meetup. It was also fantastic to catch up with Guna Petrova from Latvia, who is a key player and track organiser at Nordic Testing Days. Her outlook on testing is always refreshing and enlightening.

Without communities like TestBash, and those generated around other conferences like Let’s Test, Weekend Testing wouldn’t exist. Communities generate conversation, which lead to initiatives and plans, which lead to more communities and more conversations and deeper learning experiences. Similarly, though meetups like this, there are opportunities to develop professional relationships, which can lead to other meetups, brown bag sessions, invites to speak at conferences, or even work!

Weekend Testing Europe: Amy, Neil and Me

Later in the evening led to even more discovery and exploration of our craft (testing, beer and music). It with great surprise that I could discuss the merits of the music of Fairport Convention and Jefferson Airplane (whom, thanks to my Father, I have an appreciation of) with Michael Bolton and Neil Thompson.

But that isn’t really what we were there for. Here’s Radomir Sebek, a tester from Serbia, who works for a music production software house in Berlin. He’s playing “The Pen Game” with Michael, one of the many testing games that were going down at The Globe late into the night. That same conversation led me to be challenged on a variation of the Pen Game, this time with my observation and listening skills put to the test. I got the solution, in the end!

So here is the problem. With so many fantastic folk to talk to and learn from, you can’t really chose from them all. You pick up on different sounds and movements, explore what is interesting to you, find people you have never met before, or have had online communication with. It’s a bit like (exploratory) testing, in that you can define your conference by the actions you take, the information you gather, the people you speak to and your responses to them, and how you record them…like this.

So to the main event.

Each year, Rosie manages to attract excellent speakers to TestBash. And this year was no exception. As I mentioned in my previous post, there was no diversity in terms of gender at the 2014 conference. Not so this year, with three female speakers on the conference track. I have no details on the selection process, but I feel that the overall content, tone and message of the conference was all the better for the selections made this year.

There was also a lot to learn, from a range of experience reports, new thought leadership and science around testing, as well as technical challenges. Where TestBash is usually strong is dealing with the human element of testing, rather than drowning the attendees with technical jargon. Testing is for me very much a social discipline, as much as it is a technical discipline.

First up was Michael Bolton with “The Rapid Software Testing Guide to What you meant to say”, which looked to our use of language as a tool of our trade, and challenged many potential assumptions that could be drawn from testing behaviours. It’s my interpretation of this talk that Michael was trying to draw out the reasoning behind certain language choices in software development, and in some ways subverting their use through the prism of context driven testing. Why for example would we say automate all the testing, where we couldn’t possibly do that with development?

Up next was Iain McCowatt, with an excellent and animated discussion of the need to include intuition and the importance of tacit knowledge in our detection of bugs. Iain emphasised that socialisation and interactional expertise was an essential skill of testing. Being able to discuss and share our work and experiences appear to be key in finding bugs and communicating them effectively. It was also a great reminder to pick up the work of Harry Collins, whose writing and research contributed greatly to the themes Iain was conveying. I managed to catch up with Iain during a break, and sought his advice on combating biases in my testing. I find sometimes that because I test a lot for security, I feel that this sometimes blinds me to other considerations whilst I am testing. His insight will be invaluable in trying to balance my approach and test design processes in future.

Next up was an interesting talk about the challenges and learning gained from The Guardian’s approach to mobile testing and delivering software across multiple platforms. Sally Goble and Jonathan Hare-Winton presented a fascinating and humorous exploration of the differences and pitfalls of testing on both the iOS and Android operating systems and associated hardware. Playing on the rivalry in historic advertising campaigns between PC and Mac, and a distinctly divided audience (seemed to be more Android users than iOS, but only marginally so). This was a great talk for me, as I know very little at all about mobile application testing. The style of presentation drew more out of the audience than I expected it would, and it did not dwell too much on technical details. Great stuff!

Very excited that I’ll be able to see @sallygoble speak at #testbash. She’s a tester with a LOT of sense and skills.

After the break came the double bill of Martin Hynie and Stephen Janaway. Both talks approached the problem of organisational change and perceptions of testing and test management within development teams and businesses as a whole. Placing these two talks together was a masterstroke, as they complimented each other so well. Martin’s talk “What’s in a name? Experimenting with Testing Job Titles” focused on a social and professional science experiment. Martin found that following a change in job title and team name, to remote test, or testing; enabled his teams to have greater impact and authority within the business. He did all this under the radar, with the testers maintaining their responsibilities, whilst having a different job title. With an exciting presentation style, Martin was able to convey that maybe businesses see testing and testers as limiting and a blocker to progress. In doing so, he discovered that other teams and key stakeholders responded more positively to the alternatives. There is a lot to discover in this talk, and I won’t spoilt it further for anyone who want’s to watch the video when it comes online. Let’s just say for me that Martin’s talk it is one of the highlights of the conference.

To Stephen’s talk. For a while now, Stephen has been an inspiring member of the testing community, both personally and professionally. I was invited to speak to his team at Net-A-Porter last year, which was a fantastic opportunity. So its exciting to see how he managed to evolve into his new role as a Testing Coach, in his talk “Why I lost my job as a Test Manager and what I learned as a result”.

Organisational change is a very real challenge for testers. Stephen’s experiences here are both common, in terms of the need of testers to adapt professionally to change, but also uncommon in the approach taken by Stephen’s organisation. Rather than having overlapping development and test managers supervising the work of many people across teams, each team had its own development manager.

As a testing coach across the whole business, Stephen’s new role is to mentor the testers, enable and guide their professional development and learning, whilst not being responsible for their line management. This must have been an awesome task, reorganising the development team of a major online retailer, whilst at the same time maintaining delivery of products and services. This was an experience report beyond the normal recollection of events and dry facts, and really drove home that testers need to be able to be at the forefront of change in organisations, rather than being reactive to it.

Vernon Richards was up next, with “Myths and legends of software testing”. In 2014 Vernon blew the house down with his 99 second talk on this topic; a rapid fire list of misconceptions, musings, biases, and warnings. What Vernon did here was to distill the core of his message into an blisteringly and entertaining talk. After lunch and with everyone feeling a little full, it was the best of antidotes to wake us up.

Vernon’s talk drove home the need for testers to not only be creative in their approaches to testing, but to be wary of the fallacies and biases that can be derived from poor research, assumptions and inaccuracies. Also, looking at how to challenge the language used to describe testers and testing by non testers; such as “It’s just clicking a load of buttons” or “Anyone can do testing”. If we are to take ownership and responsibility for our craft we have to believe in our skills, and champion them to those outside testing, so that they are recognised and valued appropriately.

Maaret Pyhäjärvi came next, with “Quality doesn’t belong to the tester”. Maaret’s experiences of being the sole tester on the team, feeling responsible for quality when it seemed that no one else appeared to care resonated with me deeply. This story described how she managed approaches to testing on her team and began to build more positive relationships with the developers. In order to test sooner, and test better, Maaret elicited a collective responsibility for quality and testing, rather than taking on the burden on her own.

Matthew Heusser encouraged us to rethink our approach to regression and releases in his talk “Getting Rid of Release Testing”. This talk lead us through an approach to testing and releasing software incrementally, and becoming less reliant on the big bang “test everything” approach to release management.

Through drawing rather than slides, Matthew explained what he termed “The Swiss cheese model of risk”, where at each stage in a software release life cycle there can be different layers of testing, where there will be gaps and overlaps in coverage. It’s probably a scary approach for some, but resonates with me as working in a continuous delivery environment means that to test everything at the end would be inefficient, costly in terms of time and resources and likely not give us meaningful data. The tweet below reiterates clearly one of Matt’s main messages in a challenging and insightful talk.

Nearing the end of the main conference day leads us to Richard Bradshaw’s “Automation in testing”. I’ve never seen Richard speak before, but I have heard much about his ability to convey complex thinking in a clear and approachable way. I was not to be disappointed. Richard guided us through his evolving process of supporting testing using automation. Built up over a number of years of learning and experimentation, he described a mature and adaptable way of incorporating automation into your testing, for the right reasons – enabling the important checks that you might need to do frequently, allowing the tester to focus on exploration, learning and asking questions about the software under test. This was an inspired and entertaining talk, which engaged me in a topic that in the past has not always held my interest.

Now to the final presentation of the day, with Karen Johnson’s “The Art of asking questions”. This was hands down my favourite talk of the day. It was less of a presentation, more of a conversation with the audience. Karen’s slides were a simple guidance to invite us to flow through the discussion with her.

Karen explored with us the finer points of questioning, both of others and ourselves. Timing was a key theme, asking the right question at the right time, something I have struggled with in the past. Even more resonate with me was the idea that, quoting author Joshua Harris “The right thing at the wrong time is the wrong thing” in his book I Kissed Dating Goodbye: A New Attitude Toward Relationships and Romance

Drawing on her journalism background, Karen asked us to consider the kinds of questions we ask and how they might influence the kinds of responses we get in return. The classic, yet always useful what, where, why, who and how that will never fail you as long as you use them appropriately. After all, a lot of testing is about asking questions, and asking the right question could even prevent defects from occurring before a single line of code is written. The Q&A afterward brought many excellent questions from the audience, with Karen responding with great advice, book recommendations (see Twitter for a tonne of them) and practical suggestions to solving communication issues.

TestBash has now established a tradition of 99 second talks, led for the final time by Simon Knight. Many great folk stepped up to the stage alongside Neil, Amy and myself. Jokin Aspiazu really coined it with “If you can’t get money for conferences, ask for time. Time is valuable.” No truer thing has been said in such a short space of time!

The after party is both a chance to relax after a long day, but to engage with as many people as possible. The quite excellent and intimate bar The Mesmerist proved to be a great place to hang out and talk testing, such as with Mark Tomlinson (he of the infamous spinning cat at TestBash 2014).

Mark Tomlinson at the meetup

It’s the camaraderie and convivial atmosphere that really makes this event, year in year out. I recommend you come, make a week of it…to really let Brighton and TestBash soak in to you. You won’t regret it.

Spring has sprung on the UK testing scene once more, as it is now seven days from TestBash 2015, held each year so far in Brighton. To those of you living under a rock, TestBash is the one day conference track and two day workshop run by the good people of Ministry of Testing, and especially Rosie Sherry. You can find out more here.

This year there are some established members of the testing community speaking, such as Michael Bolton, Iain McCowatt, Stephen Janaway and Matthew Heusser. I am looking forward to seeing these guys speak again, as they are always excellent, with insights and content beyond the conventional.

If there was a criticism of TestBash 2014 was that there wasn’t a diverse range of speakers. There were no female speakers last year, where now there are three; Karen Johnson, Maaret Pyhäjärvi and Sally Goble. Whilst I have read blogs and tweets by these testers, I’ve never seen them speak before so this is going to be incredibly exciting.

There are also new speakers to TestBash, such as Richard Bradshaw and Vernon Richards.

I’ve known Richard for a few years now, and he is an inspiring and knowledgeable tester. I’ve never seen him speak before other than during a 99 second talk. He’s the first guy I would go to for information on automation. He describes his talk as ““Test Automation” = Things don’t have to be this way”.

On to Vernon Richards, whose epic 99 Second talk on Myths and Legends of Software testing has been expanded into a full blown talk. Again, I have known Vernon for a while in the community. Being isolated down in the South West of England means that I don’t always get to meet testers based and working in the London area, but Vernon has been on my radar for ages. Vernon’s 99 second talk last year earned him a huge cheer, and rightly so. This talk might turn out to be the jewel in the TestBash crown.

On to the workshops. Sadly I can’t attend the workshop day this year. With the TestBash workshops, it is your learning that is at the heart of it. The likes of John Stevenson, Simon Knight, Karen Johnson, Nicola Sedgwick and my Weekend Testing colleague Neil Studd all providing courses, it should add up to a fantastic day. Also running a workshop on BDD is Rikke Simonsen, who I had the pleasure of having lunch with last TestBash. Such a shame that I will be missing this fantastic opportunity to learn from them all. I’m definitely going to see if I can get in on that in 2016, as a learner or a trainer.

I cannot impress upon you enough the importance of TestBash in my career. I first attended in 2013. This was my first testing conference in three years, after what felt like some what of a period in the doldrums. I felt that I was coasting in my career and not doing enough to learn more, stretch myself creatively or professionally. I was just working.

A number of personal and professional events led me to attending that year, which gave me the kick up the backside that I really needed. As a result, I had my first speaking gigs in 2014. I am now speaking again at Nordic Testing Days this year, and visiting Let’s Test for the first time, running a workshop with Bill Matthews.

Sure, there are bigger conferences, with more tracks and a wider variety of talks, workshops and test labs, Some conferences are more popular with different testers, because of the variety of speakers and the depth and breadth of the content. However, what TestBash squeezes into only a few days in the compact and vibrant city of Brighton is phenomenal.

I’m also very proud to say that Brighton is sort of my home town. I grew up in a village not far away from there. This adds for me an additional pride and gratitude for the awesome effort that MOT and Rosie put into organising and running the event. As a result of the conference, and MOT as a whole, careers have been forged due to the community outreach and sponsorship of new testers so that they can attend courses and the conference for free, as well as other support. Some testers have even sponsored tickets themselves, which is hugely rewarding to the community. They should be thanked!

Two testers that are very important to me have so far benefited from this amazing community scholarship. Emma Keaveny has since moved to the UK from Ireland, secured her first testing role and along with Kim Knup have started to establish the first regular Brighton and Hove testing meetups.

The other was Danny Dainton, an ex infantry soldier, who actively pursued a career in testing after leaving the Army, and who I have the great honour of working with at New Voice Media. I really look forward to what these two fantastic testers do in the future, be it speaking themselves, or organising community events or just being able to work closely with them.

So, if you are going to TestBash next week, I look forward to seeing you there. It should be a fantastic event, full opportunities to learn and grow as a tester. If you want to talk to me, just grab me at Lean Bacon (ahem, Lean Coffee), at the queue for lunch, or at the Thursday or Friday meetups. It’s going to be EPIC!

Last week I had the great fortune of attending the London Tester Gathering, where first time speaker Mark Winteringham was leading the discussion with a great talk about mental models around testing tools. A write up and copy of his slides are here: What’s So Great about Webdriver

He was talking about how using Webdriver to automate (testing/checking…I’m not getting into that argument now) has helped him in his work as a test consultant.

However he also referred to the fact that when we use tools, whatever they are, they shape the way we think, how we behave and interact with software, solve problems and communicate. Essentially, through the use of the tool, they end up defining how we test…if we let them.

Referring to one of the highlights of Test Bash 3 – Iain McCowatt’s talk Automation: Time to change our models; (watch the video here) Mark raises the issue that we should be wary of tools, how we select them, use them to solve problems and achieve our end goals.

During the talk, Mark dares us to “Be Promiscuous” with our use of tools, to shop around, not to limit the way we work through limited use of tools, because ultimately that then leads to limited thinking. Whilst the analogy he used sparked a few laughs, he’s dead right.

Mark examined data on the number of conference talks about automation, and the number of automated testing jobs out there, and a large proportion of those are linked to specific tools – mostly Webdriver.

In itself this isn’t too worrying. Webdriver is clearly a popular tool, works well for most people and organisations that use it, and solves a lot of their testing problems. I asked Mark whether the ubiquity of Webdriver presented any dangers to us as testers, and his response was (If you’ll pardon me paraphrasing, it was a noisy room and I had drunk a couple of beers by then) if you define a tool, and eventually it will define you through its use. Essentially, that we should not let the industry or the ubiquity of a specific set of tools define us as testers, nor define our testing.

Whilst I am not a webdriver user, I do use other tools to solve some of the security testing problems that I encounter. In recent months, whilst I have been using tools such as Zed Attack Proxy and BurpSuite, I have found that my approach to testing has been limited by my ability to use these tools, rather than looking around the tool, or using them in different ways to solve problems.

Essentially, the tool was beginning to define how I tested…something to be avoided I feel. The tools that I have mentioned are great, and have a lot of useful features. However through using them, I focussed on what feedback they were providing me, rather than focussing on what was important – what I needed to do to identify vulnerabilities, understanding the underlying functionality and security of the application under test, and by trying to think like someone who wanted to undermine that application in order to protect it.

If that’s not clear, then perhaps this analogy would help – when we learn to drive for example, we are with an instructor, or a parent, in a car with a steering wheel, handbrake, accelerator, brake pedal, clutch pedal etc. It’s usually a small car, on urban roads with a bit of traffic. We build up a mental model of how to drive in our mind based on the tool we are using; the car, and the environment we are in, our local area.

Lets say then, you switch from one car to another – this one might have an automatic gear box and no clutch pedal. This then removes the need for the driver to make their own judgements about when to change gear, as the car will do it for you.

Lets take this a step further…the car has parking assistance technology, or anti collision or adaptive cruise control. These driver aids might then reduce our need to focus on the important skills of parking, or safe driving distances, maintaining a decent speed etc. We become reliant on the tool to do the job for us, rather than using our own mental models for a task to do it. Is these breeding better drivers? I’m not sure that it is.

At some point I will try to take this discussion a bit further and apply some of this learning to the security testing I have been doing. Last year, when I ran the talk ‘New Adventures in Security Testing’ I came up with this mnemonic: Cartoon Tester: EXTERMINATE (Thanks Andy for the great cartoon)

This is my personal developing mental model for security testing, and in the very near future I will work on challenging and modifying this model. Where appropriate I’ll work on sharing and discussing it.

Mark’s talk (and Iain’s) has really inspired me to think again about how I use tools effectively to solve testing problems, but also to remember that a tool, like any device, is only as good as the person controlling it.

I’ve since been invited to help run the Weekend Testing Europe chapter by Amy Phillips and Neil Studd, so keep your eyes open for future sessions! Amy is running one this week on API testing. It should be awesome. Go check out the details here and register:

During these sessions I referred to the work of security blogger Troy Hunt. He kindly let us use his website http://hackyourselffirst.troyhunt.com/ , which also forms the subject of his two courses Hack Yourself First and Hack Your API First.

Both courses work together as a fantastic way to get to grips with some tricky concepts, which are explained clearly, succinctly, and with humour.

Often these sorts of online courses can be quite boring, heavily laden with dry facts rather than useful examples and experience. Troy draws on examples in the course material, web and mobile applications, as well as real world vulnerabilities he has discovered during his work.

I can definitely recommend both of them, which are available on Pluralsight. The courses aren’t free, but they do a trial period. It’s worth investing in them if you can. Enjoy!

I’ve not blogged recently for various reasons, both personal and professional. But on the anniversary of my blog, I want to return with a more positive attitude to it after a fallow period. This is a quick blog as way of a catch up over the last few months activities (other than my professional and personal ones). It’s an opportunity to share some of the highlights of my experiences in the testing community recently, which have been warm and welcoming during some difficult times.

A few months ago I attended the inaugural Brighton Testing Meetup, catching up with some of the good folk I last met at TestBash 3. Brighton is sort of my home town, yet I have never worked there so having a foot in the pond that is the testing community there has been a great thing. We talked, we ate and drank and shared ideas. Early plans have been made for my future involvement, leading talks and discussions around some exciting testing topics. Emma Keaveny and Kim Knup are developing a vibrant new community of interest and I can’t wait to be more involved. Roll on 2015.

The community of testing is as varied and as exciting as the variety of people who work and learn within it. This is a good thing, perhaps the greatest thing about the community…and this is where the contrast lies.

The same week I went to Brighton, I also attended the latest Special Interest Group in Software Testing conference. SIGIST is organised by established, more academic people in the testing industry, on behalf of the BCS. It meets quarterly in London. There were a number of interesting topics being discussed, but it didn’t set my heart on fire. Only one or two talks out of the whole day really engaged me with the subject matter. Whilst there was the opportunity to learn from some experienced practitioners, there wasn’t the same emphasis on collaborative learning, challenging established testing paradigms and positive enquiry. It wasn’t a bad experience, it just didn’t make me more passionate about my craft, nor help me understand something new about testing. It was good however to catch up with some people who I have met before, and some who I hadn’t…but were on my radar. Namely Tonnvane Wiswell, Declan O’Riordan, Paul Gerrard, and Mike Jarred.

Another recent experience has been with some of the free, online and collaborative forums for learning and discussion that I have participated in. Firstly, Stephen Blower’s Testing Couch forum. This is a free and open Skype forum for any testers who are interested in talking about their craft. In the couple of times that I have attended, the chat has always been productive, supportive and non judgemental. Stephen makes this forum available periodically, usually every month or two. It’s a fantastic opportunity for experienced or novice testers to throw ideas around, be challenged and share thinking and learning.

Lastly, and probably my most positive experience was being a guest speaker in October’s Weekend Testing Europe forum. I was sharing my recent learning and experience in software testing, leading the attendees in an exploratory session with security as the focus. To a lot of the people during the chat, security testing was a new concept for which they had little experience or opportunities to learn. It was incredibly rewarding to be able to facilitate this session, not only on a personal level, but also to see many others taking up the challenge of securing their applications, and considering security as part of their testing.

Amy Phillips and Neil Studd have really breathed new life into Weekend Testing Europe, which had been dormant for a while. Keep an eye out for WTEU in the future, as it is a great way of keeping in touch with the testing community around the world. Be prepared to go in with eyes open, lots of questions, and a hunger to learn. All you need to do is volunteer two hours of your time on a Sunday afternoon. It sure beats watching Columbo repeats or traipsing round a garden centre.

So, that’s it for now. I’ll be blogging again soon. The Test Doctor will return!

Like a lot of teams, my team is cross functional. There are several engineers with different specialisms…systems development, performance and security, as well as testing like myself.

When we find an issues, we work on them together as a team. When we test code, we test it together as a team. We celebrate our successes and solve our problems, together, as a team.

The other day I uncovered an interesting bug. It was the sort of bug that still gave me that little buzz of excitement that I got when finding bugs earlier in my career. Not only was it a challenge to identify, replicate and describe but because it was found in a new piece of development which will really help our team solve an important problem for our customers and our service.

If as a team we can work together to solve this problem, then this benefits the customer and ultimately the business. It’s all we are about, day in day out.

The developer who is working to fix this issue feels that has identified the root cause, which has taken a lot of time, effort and frustration on his part. Research, experimentation, consulting with myself and other developers…wash, rinse, repeat…until the solution is found.

Tomorrow morning we start testing (in a VM on the developer’s machine) a pretty radical fix to the bug, which will mean a lot of testing effort for us both before we let it loose in the wild.

To be honest I can’t wait to get to grips with the solution to this problem. The testing challenge will be one of the toughest since I have been working within this team. I’m not sure yet what it will exactly entail, but the fun will be in discovering that in the days to come.

In what seems to now have been a storming comeback, the European chapter of Weekend Testing was a breath of fresh air in the learning opportunities for testers. You can find a link to the latest session here. Ably facilitated by Amy Phillips (@itjustbroke) and Neil Studd (@neilstudd) the session was dynamic and a great chance to talk with other testers in a relaxed environment. I didn’t even have to leave my house!

The main focus of the session was heuristics, how we understand, use and learn from them. There is a lot of great material on what heuristics are and how they can be used to inform and drive our testing ideas and execution. I won’t dwell too much on these areas but just hope to point you to some useful material:

Anyway, my main take away from this session was the ruts that sometimes as a tester that we might sometimes get stuck in. I chose the Constraints heuristic, utilising data type attacks upon the World Chat Clock application we were all discussing.

I found myself falling back onto what now I feel to be a bit of a party piece. I immediately decided to perform a few simple XSS and SQL Injection attacks against the application. As I expected but couldn’t be sure, was that the application’s user interface would prevent these kinds of basic security vulnerabilities from being exploited. I did ultimately find a way of injecting XSS, via OWASP Mantra, but not getting it to expose any data. The bug did however cause some interesting display and wrapping issues.

Rather than looking at the functionality, usability, accessibility and its overall purpose somehow I have begun to think the worst about the software under test before I have given myself a chance to really take the time to evaluate it critically, honestly and objectively. I immediately questioned how secure the application was before I considered any other factors.

In my work at New Voice Media, I am part of a cross functional development team, and part of a community of testing interest within the business. During this time I’ve taken onboard a lot of security testing skills, with still a lot more left to learn. It may be that I have taken these skills to heart and want to use them at any opportunity, to develop them further, to discover more about the underlying behaviour of the application under test.

Yet sometimes I feel guilty that I am not approaching the testing of software from any number of other directions, using other skills and techniques. Maybe the newer skills I have learned are higher up in my priority list in my mind before I take other approaches. So, there are of course biases at play here. I’d like to explore that further and challenge them in the future.

Perhaps this has something to do with the way I personally learn things? Early in my career everything was driven from scripts and spreadsheets. There was no impetus to learn better ways of testing, only how to get testing done faster with fewer bugs and more coverage. I was learning how to manage my testing, but not being critical of the testing I was doing, nor evaluating the testing of other people.

Now this kind of learning is the bread and butter of the testers I work with now. We learn, explore, test, check, learn some more, share, improve and the cycle continues. A much more positive way of working. It’s not without its problems, as quite rightly so, you are much more accountable for your work, justifying your choices and decisions. There is a certain level of emotional maturity that we as testers need to develop in order to sustain this cycle, be accountable, share our learning appropriately, learn well from mistakes and improve from them.

This is one of the reasons why I enjoyed Weekend Testing so much. You can’t really hide or be a silent observer. You need to get stuck in and get your hands dirty!

A couple of hours on a Sunday afternoon in the past has not been a huge cost to me, as I would only be doing a bit of housework, DIY, gardening, Scouting, sport or watching something geeky on TV. Soon though however my weekends will be taken up with the ultimate challenge of parenthood, so chances to learn with peers in a relaxed environment will become fewer and far between. More on that learning experience and how it relates to testing another time.

Weekend Testing: infinitely better and more rewarding than mowing your lawn. Thanks to Neil and Amy for running such a fun and exciting session. The same goes to the other participants for the opportunity to learn from you and the excellent conversation.

Posts navigation

I’m running a tutorial at ATD 2019

Follow me on Twitter

Dan Billing

I'm a software test engineer of 17 years, and recently I decided to go it alone as a consulting tester, founding my company The Test Doctor Limited.
I love testing and all its wondrous variety. I like to help others become better testers by attending events, speaking, blogging and giving training.
Most of my current work focuses on testing strategy across the whole of the clinical trials suite that we build. This includes any kind of testing, from UI, API, performance, security, mobile etc. Whatever needs to happen.
I'm also building on the training, coaching and learning I've picked up elsewhere, and bringing that into my new team.
I enjoy running workshops and speaking, especially in the technical testing and security space; and to a lesser extent the psychology of what testers do.
Hopefully, It'll make me a better tester too!