Jekyll2018-03-19T07:03:31+00:00http://testersdigest.mehras.net/Tester’s DigestA weekly source of software testing newsdmehraTester’s Digest #51: Visual Testing2018-03-18T00:00:00+00:002018-03-18T00:00:00+00:00http://testersdigest.mehras.net/2018/03/18/testers-digest-51-visual-testing<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #51 - March 18, 2018</p>
<p>Continuing coverage of web UI testing, we focus on visual testing / validation: what it is (just pixel comparison of screenshots, really, but with a lot of gotchas), how to do it, choice of tools and service providers. This became a long issue as I kept digging into the topic. There appear to be two levels of visual testing, with one set of tools geared more toward end-to-end tests that run in staging environment (Applitools Eyes, Percy, Backtrac), and another targeting developers at unit test level (React Storybook, Jest image snapshots, etc).</p>
<h1 id="topic-visual-testing">Topic: Visual Testing</h1>
<p>Distinction between functional testing of the UI and visual testing:</p>
<p><a href="https://saucelabs.com/blog/review-of-visual-vs-functional-testing-with-pdiff-and-applitools">https://saucelabs.com/blog/review-of-visual-vs-functional-testing-with-pdiff-and-applitools</a></p>
<p>Why do visual testing, by Applitools but not specific to their product. It’s the basic idea of recording screenshots from functional tests, doing automated image comparison with last round’s screenshots, and serving up the changed ones to a human for validation.</p>
<p><a href="http://testautomation.applitools.com/post/152615349182/why-visual-testing-and-agile-are-a-perfect-fit">http://testautomation.applitools.com/post/152615349182/why-visual-testing-and-agile-are-a-perfect-fit</a></p>
<p><a href="http://testautomation.applitools.com/post/153726980802/the-roi-of-visual-testing">http://testautomation.applitools.com/post/153726980802/the-roi-of-visual-testing</a></p>
<p>What’s hard about diffing screenshots for visual regression testing? Post from another service provider who promises to do it better (Backtrac).</p>
<p><a href="https://backtrac.io/blog/creating-screenshot-what-can-go-wrong">https://backtrac.io/blog/creating-screenshot-what-can-go-wrong</a></p>
<p>The third popular provider, Percy, writes about using their own product to validate a large scale restyling of their site.</p>
<p><a href="https://blog.percy.io/redesigning-with-confidence-d11799845ecb">https://blog.percy.io/redesigning-with-confidence-d11799845ecb</a></p>
<p>More detail on what it looks like to test with Applitools Eyes. This is written by their evangelist, using a Calculator app as an example:</p>
<p><a href="https://hackernoon.com/testing-your-frontend-code-part-v-visual-testing-935864cfb5c7">https://hackernoon.com/testing-your-frontend-code-part-v-visual-testing-935864cfb5c7</a></p>
<p>This satisfied user of Applitools built a dynamic baseline by comparing staging and production versions of the app pages via Selenium tests. Their approach aims to solve the problem of needing to thumbs up/down on lots of changes to manually re-record the baseline.</p>
<p><a href="https://engineering.datorama.com/dynamic-visual-testing-at-scale-how-we-automate-testing-of-our-analytics-dashboards-db72e261ad75">https://engineering.datorama.com/dynamic-visual-testing-at-scale-how-we-automate-testing-of-our-analytics-dashboards-db72e261ad75</a></p>
<p>How developers can incorporate visual testing into Test Driven Development, with React Storybook. Tests run automated, output gets checked manually. The post doesn’t cover how this would integrate into CI, perhaps with a manual build stage?..</p>
<p>&lt;https://blog.hichroma.com/visual-test-driven-development-aec1c98bed87</p>
<p>Nice description of why you shouldn’t compare complete screenshots of your website/app pages from someone who did it that way at first, then moved to component-level isolated screenshots, advocates BackstopJS.</p>
<p><a href="https://medium.com/@philgourley/making-visual-regression-useful-acfae27e5031">https://medium.com/@philgourley/making-visual-regression-useful-acfae27e5031</a></p>
<p>Good analysis of what snapshot tests are good for, at unit test level (using Jest image snapshots):</p>
<p><a href="https://benmccormick.org/2016/09/19/testing-with-jest-snapshots-first-impressions/">https://benmccormick.org/2016/09/19/testing-with-jest-snapshots-first-impressions/</a></p>
<p>Reflection on how visual testing fits into the work process of a development team, including code review.</p>
<p><a href="https://engineering.klarna.com/improving-communication-and-confidence-with-visual-snapshot-testing-b04154c3aaf0">https://engineering.klarna.com/improving-communication-and-confidence-with-visual-snapshot-testing-b04154c3aaf0</a></p>
<p>Lists of (mostly free open source) visual validation tools:</p>
<p><a href="https://www.joecolantonio.com/2017/02/02/top-21-free-visual-validation-tools-testers/">https://www.joecolantonio.com/2017/02/02/top-21-free-visual-validation-tools-testers/</a></p>
<p><a href="https://github.com/mojoaxel/awesome-regression-testing/blob/master/README.md">https://github.com/mojoaxel/awesome-regression-testing/blob/master/README.md</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Worth learning: debug Javascript code with console tricks beyond console.log()</p>
<p><a href="https://medium.com/appsflyer/10-tips-for-javascript-debugging-like-a-pro-with-console-7140027eb5f6">https://medium.com/appsflyer/10-tips-for-javascript-debugging-like-a-pro-with-console-7140027eb5f6</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraVisual testing - what it is (just pixel comparison of screenshots, really, but with a lot of gotchas), how to do it, choice of tools and service providers.Tester’s Digest #50: Front End Testing2018-03-11T00:00:00+00:002018-03-11T00:00:00+00:00http://testersdigest.mehras.net/2018/03/11/testers-digest-50-front-end-testing<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #50 - March 11, 2018</p>
<p>This post and the next couple will be on testing web UI, from front-end components to end-to-end workflows. Today’s focus is on Javascript / React. An earlier post on UI test automation with some still-relevant resources can be found at:</p>
<p><a href="http://testersdigest.mehras.net/2017/07/30/testers-digest-26-ui-test-automation.html">http://testersdigest.mehras.net/2017/07/30/testers-digest-26-ui-test-automation.html</a></p>
<h1 id="topic-front-end-testing">Topic: Front End Testing</h1>
<p>Which testing frameworks are used with today’s Javascript? Mocha, Jasmine, Jest, Cucumber JS. (This answer probably became obsolete as I typed it, given JS community’s rate of change.)</p>
<p><a href="https://gojko.net/2018/02/25/javascript-testing-tools.html">https://gojko.net/2018/02/25/javascript-testing-tools.html</a></p>
<p>Nice detailed overview of different testing layers and frameworks for Javascript front end. TL;DR; Use Jest for unit and integration tests and TestCafe for UI tests.</p>
<p><a href="https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3">https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3</a></p>
<p>How one team selected a front-end testing framework, settling on Nightwatch.js. Our web development team at Quid took the journey from WebdriverIO to TestCafe to Cypress - we should blog about that…</p>
<p><a href="http://adventuresinqa.com/2017/09/19/nightwatch/">http://adventuresinqa.com/2017/09/19/nightwatch/</a></p>
<p>Use cases and design considerations for a date picker widget, food for thought in testing UI elements:</p>
<p><a href="https://www.smashingmagazine.com/2017/07/designing-perfect-date-time-picker/">https://www.smashingmagazine.com/2017/07/designing-perfect-date-time-picker/</a></p>
<p>And a couple of handy tools. This post covers Chrome extensions for testing web page link validity and providing problematic inputs and mocked data for input forms.</p>
<p><a href="https://michaldymek.me/essential-testing-tools/">https://michaldymek.me/essential-testing-tools/</a></p>
<p>Carte Blanche, integrated fuzz testing for front-end components. Warning: hasn’t been updated in 2 years; but works with React.</p>
<p><a href="https://github.com/carteb/carte-blanche">https://github.com/carteb/carte-blanche</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>HTTP status codes explained with cat or dog GIFs, good for lulz, great as a memory aide:</p>
<p><a href="https://http.cat/">https://http.cat/</a></p>
<p><a href="https://httpstatusdogs.com/">https://httpstatusdogs.com/</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraFrameworks and handy tools for testing front end components and end-to-end flows, with focus on Javascript / React.Tester’s Digest #49: Shift-Left2018-03-04T00:00:00+00:002018-03-04T00:00:00+00:00http://testersdigest.mehras.net/2018/03/04/testers-digest-49-shift-left<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #49 - March 4, 2018</p>
<p>Shift-left is a popular term in testing these days. What does it mean to shift testing earlier in the software development process, and why is it beneficial?</p>
<h1 id="topic-shift-left">Topic: Shift-Left</h1>
<p>Shift-left/right with some specifics of who does what when:</p>
<p><a href="https://techbeacon.com/shift-your-testing-how-increase-quality-not-anxiety">https://techbeacon.com/shift-your-testing-how-increase-quality-not-anxiety</a></p>
<p>A PM’s take on shift-right / shift-left views testing holistically, without separating usability feedback from classic validation testing.</p>
<p><a href="https://blog.hiptest.net/2015/06/26/shift-left-and-shift-right-the-testing-swing/">https://blog.hiptest.net/2015/06/26/shift-left-and-shift-right-the-testing-swing/</a></p>
<p>Nice deck on the problem of “we have no time for testing!” and amelioration ideas with shifting testing earlier in the process, risk-based testing, and devops. No depth in the slides, just bullet points.</p>
<p><a href="https://www.slideshare.net/AlexSchwartz1/dev-opsberlin-ignitehelpwehavenomoretimefortestingslides">https://www.slideshare.net/AlexSchwartz1/dev-opsberlin-ignitehelpwehavenomoretimefortestingslides</a></p>
<p>The value of early inclusion of testers on dev projects is in driving clarification at requirements gathering / definition stage; wireframes testing at design stage; bug discovery at development stage; system testing (and nothing more, since other levels of testing are already done!) post integration. Early involvement also builds credibility and avoids the problem of having no time to test when the feature is all coded up.</p>
<p><a href="https://chroniclesoftesting.blogspot.co.id/2017/05/when-should-tester-engage-in-testing.html">https://chroniclesoftesting.blogspot.co.id/2017/05/when-should-tester-engage-in-testing.html</a></p>
<p>Ask questions early to prevent problems later:</p>
<p><a href="https://www.testingexcellence.com/software-testing-ask-questions/">https://www.testingexcellence.com/software-testing-ask-questions/</a></p>
<p>The impact of Quality Driven Development process for one organization. By QDD they mean BDD with automated tests running in dev environment, fixing issues on the fly, and encoding the definition of done as a demo by automated test that drives the UI. This team saw good outcomes: “streamlined our testing process, made us more agile than ever before, raised the quality of our products, and given us an increased awareness of customer satisfaction”.</p>
<p><a href="https://www.stickyminds.com/article/impact-quality-driven-development">https://www.stickyminds.com/article/impact-quality-driven-development</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Adding games and gamification techniques to meetings and other workplace activities, a tester’s perspective:</p>
<p><a href="https://dojo.ministryoftesting.com/lessons/gamifying-your-software-testing-career-workplace-part-2">https://dojo.ministryoftesting.com/lessons/gamifying-your-software-testing-career-workplace-part-2</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraShift-left is a popular term in testing these days. What does it mean to shift testing earlier in the software development process, and why is it beneficial?Tester’s Digest #48: Load Testing2018-02-25T00:00:00+00:002018-02-25T00:00:00+00:00http://testersdigest.mehras.net/2018/02/25/testers-digest-48-load-testing<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #48 - February 25, 2018</p>
<p>Tester’s Digest is back after a flu break. We look at the classical QA task of load testing, with some modern twists - should developers do it? should you load test in production? how can it all go wrong?</p>
<h1 id="topic-load-testing">Topic: Load Testing</h1>
<p>Should developers write their own load tests, rather than have QA or another team do it for them? This post votes yes:</p>
<p><a href="https://engineering.klarna.com/four-reasons-developers-should-write-their-own-load-tests-fac74c1be9f1#.sizpzib0g">https://engineering.klarna.com/four-reasons-developers-should-write-their-own-load-tests-fac74c1be9f1#.sizpzib0g</a></p>
<p>How you can write load tests wrong: 1) make them too short; 2) ignore anomalies; 3) reuse test data (or rather, fail to disable caching); 4) only load test under happy path conditions with no failures.</p>
<p><a href="https://engineering.klarna.com/four-load-testing-mistakes-developers-love-to-make-68b443f7e8a2#.3ea8he5fu">https://engineering.klarna.com/four-load-testing-mistakes-developers-love-to-make-68b443f7e8a2#.3ea8he5fu</a></p>
<p>Additional uses of load tests: to help reproduce a sporadic performance issue, or to uncover app slowness based on user’s location.</p>
<p><a href="https://www.stickyminds.com/article/using-load-testing-tools-more-just-load-testing">https://www.stickyminds.com/article/using-load-testing-tools-more-just-load-testing</a></p>
<p>Load testing of websites with Vegeta tool and test data in a Python Pandas dataframe:</p>
<p><a href="https://serialized.net/2017/06/load-testing-with-vegeta-and-python/">https://serialized.net/2017/06/load-testing-with-vegeta-and-python/</a></p>
<p>Website Speed Test tool, built on top of the well known WebPagetest, analyzes load time of images on your website and makes improvement suggestions, complete with optimized images you can download and use:</p>
<p><a href="https://www.smashingmagazine.com/2017/07/website-speed-test-image-analysis-tool/">https://www.smashingmagazine.com/2017/07/website-speed-test-image-analysis-tool/</a></p>
<p>Open source tools for load and stress testing (Fiddler, JMeter, Locust, Taurus, Gatling, Siege and more):</p>
<p><a href="https://www.joecolantonio.com/2017/07/18/open-source-performance-testing-tools/">https://www.joecolantonio.com/2017/07/18/open-source-performance-testing-tools/</a></p>
<p>On different ways of setting up a load testing environment, from testing in production to rolling your own:</p>
<p><a href="https://www.stickyminds.com/article/six-tips-building-better-load-testing-environment">https://www.stickyminds.com/article/six-tips-building-better-load-testing-environment</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Worth learning: Introduction to distributed systems. Per Lamport, 1987, “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable”. These days, there is no other kind of systems…</p>
<p><a href="https://caitiem.com/2017/09/07/getting-started-with-distributed-systems/">https://caitiem.com/2017/09/07/getting-started-with-distributed-systems/</a></p>
<p><a href="https://github.com/aphyr/distsys-class">https://github.com/aphyr/distsys-class</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraWe look at the classical QA task of load testing, with some modern twists.Tester’s Digest #47: Writing Better Code2018-02-11T00:00:00+00:002018-02-11T00:00:00+00:00http://testersdigest.mehras.net/2018/02/11/testers-digest-47-writing-better-code<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #47 - February 11, 2018</p>
<p>You can’t test the quality in, they say. So how do we get to build software at a better quality level? We examine some techniques for writing better code: code reviews (duh), writing good commit messages and error messages, pairing and mobbing approaches. Tell your developers, and keep in mind that test code is code, so let’s practice what we preach.</p>
<h1 id="topic-writing-better-code">Topic: Writing Better Code</h1>
<p>Tips for better code review process. The romance analogy is cheesy (yet suitable for the upcoming Valentine’s day!), don’t let it throw you off, it gets good later on.</p>
<p><a href="https://mtlynch.io/human-code-reviews-1/">https://mtlynch.io/human-code-reviews-1/</a></p>
<p>How to write better commit messages, which then help code review and make later “git blame” useful:</p>
<p><a href="https://medium.com/@felixclack/writing-great-commit-messages-for-better-code-review-70b21dac5788">https://medium.com/@felixclack/writing-great-commit-messages-for-better-code-review-70b21dac5788</a></p>
<p>How to write good error messages:</p>
<p><a href="https://flipboard.com/@rosiesherry/ministry-of-testing-87epnnl5y/how-to-write-a-perfect-error-message/a-IThmjYRsRYiyvIh4tFCzlA%3Aa%3A2636205-bc6cb053bf%2Fuxplanet.org">https://flipboard.com/@rosiesherry/ministry-of-testing-87epnnl5y/how-to-write-a-perfect-error-message/a-IThmjYRsRYiyvIh4tFCzlA%3Aa%3A2636205-bc6cb053bf%2Fuxplanet.org</a></p>
<p>How to build Web APIs for success, principles from Salesforce, including “Trust in Acceptance Tests” (BDD style) and “Log, Monitor and Alert” – I couldn’t agree more.</p>
<p><a href="https://engineering.salesforce.com/setting-up-a-web-api-for-success-ff039f76d322">https://engineering.salesforce.com/setting-up-a-web-api-for-success-ff039f76d322</a></p>
<p>Nice little rant imploring developers to deliver software that’s challenging for testers to find bugs in:</p>
<p><a href="https://blog.iain.xyz/2017/06/your-test-team-is-not-a-safety-blanket.html">https://blog.iain.xyz/2017/06/your-test-team-is-not-a-safety-blanket.html</a></p>
<p>Pair programming has promise of delivering a higher quality product with increased velocity in the long run. This article is a how-to guide:</p>
<p><a href="https://medium.com/@weblab_tech/pair-programming-guide-a76ca43ff389">https://medium.com/@weblab_tech/pair-programming-guide-a76ca43ff389</a></p>
<p>While this paper outlines costs (15% increase in development time) and benefits: “improves design quality, reduces defects, reduces staffing risk, enhances technical skills, improves team communications and is considered more enjoyable”</p>
<p><a href="https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia.PDF">https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia.PDF</a></p>
<p>Tester-developer pairing technique promises benefits in learning on both sides, earlier bug discovery, and more maintainable automated tests:</p>
<p><a href="https://dojo.ministryoftesting.com/lessons/pairing-with-developers-a-guide-for-testers">https://dojo.ministryoftesting.com/lessons/pairing-with-developers-a-guide-for-testers</a></p>
<p>Mobbing technique with both programmers and testers participating can “turn programmers into better testers, and improve any testers capabilities”.</p>
<p><a href="https://dojo.ministryoftesting.com/lessons/mob-testing-an-introduction-experience-report">https://dojo.ministryoftesting.com/lessons/mob-testing-an-introduction-experience-report</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Worth learning: On more effective ways of presenting test information to others, based on data visualization principles from Edward Tufte, the god of viz:</p>
<p><a href="https://blog.gurock.com/youve-got-to-see-this/">https://blog.gurock.com/youve-got-to-see-this/</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraHow do we get build software at a better quality level? You can't test the quality in, they say, so what can you do? Code reviews, pairing techniques, and more.Tester’s Digest #46: Technical Debt2018-02-04T00:00:00+00:002018-02-04T00:00:00+00:00http://testersdigest.mehras.net/2018/02/04/testers-digest-46-technical-debt<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #46 - February 4, 2018</p>
<p>It’s been nearly a year since we last looked at technical debt, and the new content written about the issue has mounted along with the debt itself. This is our new look at the problem of technical debt, its subtypes, measurement, and ways of paying it off. The earlier take is available in Tester’s Digest archive:</p>
<p><a href="http://testersdigest.mehras.net/2017/03/19/testers-digest-7-technical-debt.html">http://testersdigest.mehras.net/2017/03/19/testers-digest-7-technical-debt.html</a></p>
<h1 id="topic-technical-debt">Topic: Technical Debt</h1>
<p>Kinds of technical debt, including defect debt, and some ideas for quantifying the costs of carrying such debt:</p>
<p><a href="https://blog.cloudymusings.com/a-lexicon-of-software-development-debt-6d88524f0a19">https://blog.cloudymusings.com/a-lexicon-of-software-development-debt-6d88524f0a19</a></p>
<p>Bug debt! “Unresolved bugs are like the undeliverable mail of our day—a one-way communication without a recipient.”</p>
<p><a href="https://www.stickyminds.com/article/clean-your-bug-tracker-and-keep-numbers-manageable">https://www.stickyminds.com/article/clean-your-bug-tracker-and-keep-numbers-manageable</a></p>
<p>Feature flags are considered technical debt, for very good reasons:</p>
<p><a href="https://dzone.com/articles/feature-toggles-are-one-worst">https://dzone.com/articles/feature-toggles-are-one-worst</a></p>
<p>The big reason behind technical debt: we have systems with technical debt because those systems work – that is, they appear to be working well enough, since the “working” part is visible, and the “debt” part is not.</p>
<p><a href="https://medium.com/@GeneHughson/dealing-with-technical-debt-like-we-mean-it-155a98a39f1c">https://medium.com/@GeneHughson/dealing-with-technical-debt-like-we-mean-it-155a98a39f1c</a></p>
<p>Why isn’t anyone doing anything about the tech debt? Let’s hope the situation at your workplace is not as dire as in this fictional bureaucracy:</p>
<p><a href="https://hackernoon.com/were-drowning-in-tech-debt-why-isn-t-anyone-listening-f4269cb5cc40">https://hackernoon.com/were-drowning-in-tech-debt-why-isn-t-anyone-listening-f4269cb5cc40</a></p>
<p>PagerDuty on measuring technical debt with incident related data. While this is obviously tooting their own horn, the idea appears sound. “For example, if your MTTR for incidents related to a certain program is higher than your average MTTR, there’s a good chance the program in question is generating technical debt. Similarly, if servers running one type of operating system account for a disproportionate number of alerts, there’s probably a code or configuration flaw at play. That’s a technical debt you can address.”</p>
<p><a href="https://www.pagerduty.com/blog/technical-debt/">https://www.pagerduty.com/blog/technical-debt/</a></p>
<p>When taking on technical debt (on dev or test side), have a specific plan for how you will pay it off:</p>
<p><a href="https://testwithnishi.com/2017/09/07/paying-off-the-technical-debt-in-your-agile-projects/">https://testwithnishi.com/2017/09/07/paying-off-the-technical-debt-in-your-agile-projects/</a></p>
<p>One approach to dipping into your technical debt… spin the wheel!</p>
<p><a href="https://goodenoughsoftware.net/2016/11/30/wheel-of-technical-debt/">https://goodenoughsoftware.net/2016/11/30/wheel-of-technical-debt/</a></p>
<p>How LinkedIn paused new development for 2 months to pay off its tech debt in 2011 - light on detail, big on inspiration:</p>
<p><a href="https://www.linkedin.com/pulse/when-your-tech-debt-comes-due-kevin-scott/">https://www.linkedin.com/pulse/when-your-tech-debt-comes-due-kevin-scott/</a></p>
<p>The system complexity angle on managing technical debt: have points of flexibility in the right places in your software (not everywhere), minimize dependencies, refactor for velocity, throw away prototype code and low-performing features, build culture of testing and code review.</p>
<p><a href="https://hackernoon.com/managing-technical-debt-1806424e7d40">https://hackernoon.com/managing-technical-debt-1806424e7d40</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Worth learning: how to be a wizard. Well, that’s the title anyway, but this slide deck from Stripe’s engineer Julia Evans is amazing (take it from the person who hates slide decks). It covers the learning process of a future great engineer (or tester!)- what do you read? what do you try? how do you ask questions? how do you debug? how do you design? how do you develop understanding?</p>
<p><a href="https://www.slideshare.net/JuliaEvans8/so-you-want-to-be-a-wizard-73101468">https://www.slideshare.net/JuliaEvans8/so-you-want-to-be-a-wizard-73101468</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraA new look at the problem of technical debt, its subtypes, measurement, and ways of paying it off.Tester’s Digest #45: Security Testing2018-01-28T00:00:00+00:002018-01-28T00:00:00+00:00http://testersdigest.mehras.net/2018/01/28/testers-digest-45-security-testing<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #45 - January 28, 2018</p>
<p>We covered security risks in the last issue, and will continue the topic into security testing.</p>
<h1 id="topic-security-testing">Topic: Security Testing</h1>
<p>Quality vs security. Some questionable statements there such as “quality is binary – the software either works or it doesn’t”, but good links to resources on weaknesses. Mentions fuzz testing as a useful technique to find security holes.</p>
<p><a href="https://www.synopsys.com/blogs/software-security/does-software-quality-equal-software-security/">https://www.synopsys.com/blogs/software-security/does-software-quality-equal-software-security/</a></p>
<p>What it’s like to be a bug hunter in the security space:</p>
<p><a href="https://blog.cobalt.io/learn-through-play-following-the-path-of-a-bug-hunter-8ed64092aa7b">https://blog.cobalt.io/learn-through-play-following-the-path-of-a-bug-hunter-8ed64092aa7b</a></p>
<p>Yelp’s public bug bounty program finds some hard-to-find critical security vulnerabilities, while the earlier private program weeded out the common ones:</p>
<p><a href="https://engineeringblog.yelp.com/2016/12/100-days-public-bug-bounty-program.html">https://engineeringblog.yelp.com/2016/12/100-days-public-bug-bounty-program.html</a></p>
<p>Automating security acceptance tests in a BDD framework, using OWASP ZAP:</p>
<p><a href="https://opencredo.com/automating-your-security-acceptance-tests/">https://opencredo.com/automating-your-security-acceptance-tests/</a></p>
<p>Security patterns in web apps, and a Rails specific tool called SPACE (Security PAttern CheckEr)
that promises to find bugs if the developer defines a lightweight mapping from code to patterns:</p>
<p><a href="https://blog.acolyer.org/2017/02/07/finding-security-bugs-in-web-applications-using-a-catalog-of-access-control-patterns/">https://blog.acolyer.org/2017/02/07/finding-security-bugs-in-web-applications-using-a-catalog-of-access-control-patterns/</a></p>
<p>DIY pen testing:</p>
<p><a href="https://highon.coffee/blog/penetration-testing-tools-cheat-sheet/">https://highon.coffee/blog/penetration-testing-tools-cheat-sheet/</a></p>
<p>OWASP mobile security testing guide defines itself as a comprehensive manual and certainly is, with sections on general code quality and cryptography in mobile apps, common attacks, memory corruption bugs, auth architectures, network communication testing, and specifics of Android and iOS. It is, in a word, amazing, well maintained and current.</p>
<p><a href="https://github.com/OWASP/owasp-mstg">https://github.com/OWASP/owasp-mstg</a></p>
<p>If you need to test HTTPS clients implementing the common TLS encryption protocol, Yelp gives you tlspretense-service tool, and Netflix open sourced bettertls which specifically targets name constraints for HTTPS clients:</p>
<p><a href="https://engineeringblog.yelp.com/2015/05/https-client-testing-made-easy.html">https://engineeringblog.yelp.com/2015/05/https-client-testing-made-easy.html</a></p>
<p><a href="http://techblog.netflix.com/2017/04/bettertls-name-constraints-test-suite.html">http://techblog.netflix.com/2017/04/bettertls-name-constraints-test-suite.html</a></p>
<p>Security Monkey is another tool from Netflix, this one monitors AWS changes and alerts on security problems, also released for Google Cloud Platform:</p>
<p><a href="http://techblog.netflix.com/2017/03/netflix-security-monkey-on-google-cloud.html">http://techblog.netflix.com/2017/03/netflix-security-monkey-on-google-cloud.html</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>An interesting take on the limitations of Chaos Engineering:</p>
<p><a href="https://medium.com/production-ready/the-limitations-of-chaos-engineering-2a74816c0df3">https://medium.com/production-ready/the-limitations-of-chaos-engineering-2a74816c0df3</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraWe covered security risks in the last issue, and will continue the topic into security testing.Tester’s Digest #44: Security Risks2018-01-22T00:00:00+00:002018-01-22T00:00:00+00:00http://testersdigest.mehras.net/2018/01/22/testers-digest-44-security-risks<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #44 - January 22, 2018</p>
<p>In the aftermath of Spectre and Meltdown, let’s talk about security risks. A lot of my links in this issue are to The Morning Paper blog because its awesomeness is unsurpassed. Many modern security vulnerabilities are based on observable side effects, and Adrian Colyer has covered various kinds in his writeups.</p>
<h1 id="topic-security-risks">Topic: Security Risks</h1>
<p>The new OWASP list of top security risks is out! See the 2017 PDF for the application security, and 2016 for mobile. This is a great resource for testers and developers alike.</p>
<p><a href="https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project">https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project</a></p>
<p>How does Meltdown attack work, really? The Morning Paper tells all. In a nutshell, the attacker’s makes an access to the memory location whose content they wish to know, but are not authorized to access; however, just before that access, the code throws an exception. The plucky CPU will speculatively execute the post-exception code instruction anyway, as a performance optimization (it didn’t know you’d be throwing an exception, and optimized for the happy path). After the exception, the speculative execution path gets rolled back, but leaves a side effect - values in the cache. Now the attacker can do Flush+Reload, iterating over the pages (inside the exception handler) until it finds the cache hit and can recover the secret.</p>
<p><a href="https://blog.acolyer.org/2018/01/15/meltdown/">https://blog.acolyer.org/2018/01/15/meltdown/</a></p>
<p>What about Spectre? It’s like Meltdown’s scarier sibling, since it can achieve similar outcome without needing to trigger an exception, by exploiting speculative execution of branched code. My favorite blog has it covered as well, with a code sample in Javascript if you’d like to run your very own exploit.</p>
<p><a href="https://blog.acolyer.org/2018/01/16/spectre-attacks-exploiting-speculative-execution/">https://blog.acolyer.org/2018/01/16/spectre-attacks-exploiting-speculative-execution/</a></p>
<p>Required reading for all your friends who are front-end developers - here’s how to (not) potentially leak all the sensitive information from your user-data-collecting webpage via nefarious npm modules. A fun read.</p>
<p><a href="https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5">https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5</a></p>
<p>A look at the attacks on databases that use access leakage, or communication volume leakage (which all known systems suffer from!); also known as “side channel attacks”, these approaches latch on to the information revealed by the communication pattern itself, even though its contents are encrypted. Think of the thin vs thick envelope received by the college applicants - the postman knows whether you got admitted… The obvious defense is padding of the records (ie: everybody gets a fat envelope stuffed with blank pages), but as it is expensive, most existing practical systems are non-storage-inflating.</p>
<p><a href="https://blog.acolyer.org/2016/11/16/generic-attacks-on-secure-outsourced-databases/">https://blog.acolyer.org/2016/11/16/generic-attacks-on-secure-outsourced-databases/</a></p>
<p>Systems built on property revealing encryption promise to allow private unencrypted data to stay only in the user’s browser, while all communication with the server and server-side processing is encrypted (while still allowing operations such as search and sharing by the server). However, there are types of attacks this setup is open to. Observing access patterns can be enough to infer private information, for example: if the user’s shopping cart (items and prices) is encrypted, but the user’s search history on the shopping website is not, it’s a good guess that the shopping cart consists of recently visited items.</p>
<p><a href="https://blog.acolyer.org/2016/11/15/breaking-web-applications-built-on-top-of-encrypted-data/">https://blog.acolyer.org/2016/11/15/breaking-web-applications-built-on-top-of-encrypted-data/</a></p>
<p>All your base are belong to us… with a simple USB stick inserted into a locked (!) PC - Raspberry Pi-based PoisonTap by the white-hat hacker Samy Kamkar. The malicious code is hidden in the browser cache (you clean that out regularly don’t you? :-)) and the preconditions are Wi-Fi access and at least one open browser tab that periodically downloads updates. Once PoisonTap is unplugged, the backdoor remains in the browser cache, giving the hacker a way in using standard exploits.</p>
<p><a href="https://www.wired.com/2016/11/wickedly-clever-usb-stick-installs-backdoor-locked-pcs/">https://www.wired.com/2016/11/wickedly-clever-usb-stick-installs-backdoor-locked-pcs/</a></p>
<p>Code signing is a standard technique that you “just do” to validate integrity of software updates, right? Not in cars, apparently:</p>
<p><a href="https://www.wired.com/2016/09/tesla-responds-chinese-hack-major-security-upgrade/">https://www.wired.com/2016/09/tesla-responds-chinese-hack-major-security-upgrade/</a></p>
<p>Weaknesses of password managers, types of attack against them, and possible defenses (secure filling). Based on this, 1Password is the better choice, with LastPass a possible alternative (they both must have improved since 2014 when studied):</p>
<p><a href="https://blog.acolyer.org/2017/02/06/password-managers-attacks-and-defenses/">https://blog.acolyer.org/2017/02/06/password-managers-attacks-and-defenses/</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>The Hawaii incident of erroneously triggered missile warning has its roots in the UX design of the selection screen. What would you have clicked, the DRILL link, or the link right under TEST MESSAGE? Don Norman, the author of “The Design of Everyday Things”, walks through the many problems with Hawaii UI and the proper design of systems with “test mode”.</p>
<p><a href="https://www.fastcodesign.com/90157153/don-norman-what-went-wrong-in-hawaii-human-error-nope-bad-design">https://www.fastcodesign.com/90157153/don-norman-what-went-wrong-in-hawaii-human-error-nope-bad-design</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraIn the aftermath of Spectre and Meltdown, let's talk about security risks.Tester’s Digest #43: Debugging Tales2018-01-13T00:00:00+00:002018-01-13T00:00:00+00:00http://testersdigest.mehras.net/2018/01/13/testers-digest-43-debugging-tales<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #43 - January 13, 2018</p>
<p>Tester’s Digest is back after a break for the holiday season, flu season, and performance review season! This issue is full of debugging stories. Debugging skills tend to be hard won through apprenticeship and trial-and-error, so I’m always excited when I come across writeups of specific situations to learn from.</p>
<h1 id="topic-debugging-tales">Topic: Debugging Tales</h1>
<p>From debugging the Joyent outage of 5/27/2014, to thinking about the art and culture of debuggability, this is a great deck:</p>
<p><a href="https://www.slideshare.net/bcantrill/debugging-under-fire-keeping-your-head-when-systems-have-lost-their-mind">https://www.slideshare.net/bcantrill/debugging-under-fire-keeping-your-head-when-systems-have-lost-their-mind</a></p>
<p>Also from Joyent (as acquired by Samsung), a deck on finding unusual pathologies (“zebras not horses”) in the data path:</p>
<p><a href="https://www.slideshare.net/bcantrill/zebras-all-the-way-down-the-engineering-challenges-of-the-data-path">https://www.slideshare.net/bcantrill/zebras-all-the-way-down-the-engineering-challenges-of-the-data-path</a></p>
<p>A story of a debugging deep dive, starting with investigation into a sudden traffic drop at the load balancer, then going into the nitty gritty of CPU and memory usage on the haproxy machines.</p>
<p><a href="https://blog.booking.com/troubleshooting-a-journey-into-the-unknown.html">https://blog.booking.com/troubleshooting-a-journey-into-the-unknown.html</a></p>
<p>An instructive story of debugging a “flaky” test:</p>
<p><a href="http://blog.jgc.org/2013/07/your-test-suite-is-trying-to-tell-you.html">http://blog.jgc.org/2013/07/your-test-suite-is-trying-to-tell-you.html</a></p>
<p>Debugging a race condition in the Python Queue class, “a tragedy of deadlocks and despair”:</p>
<p><a href="https://codewithoutrules.com/2017/08/16/concurrency-python/">https://codewithoutrules.com/2017/08/16/concurrency-python/</a></p>
<p>Debugging memory leaks in Ruby code:</p>
<p><a href="https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby">https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby</a></p>
<p>Debugging a data corruption issue in RavenDB: I appreciate not only the deep dive into the debugging process, but thoughts around prioritization (“All development ceases until [memory damage bug] is found”) and postmortem notes.</p>
<p><a href="https://ayende.com/blog/180481/production-postmortem-data-corruption-a-view-from-inside-the-sausage">https://ayende.com/blog/180481/production-postmortem-data-corruption-a-view-from-inside-the-sausage</a></p>
<p>This debugging story is on its own level of awesome: guy decides to monitor his laptop’s resource utilization so he can be warned when, say, too many Chrome tabs are eating up too much memory (I need that!), but after setting up Prometheus, runs into a segfault crash. The investigation involves, among other things. a heat gun and building 32 kernels. Not to be missed.</p>
<p><a href="https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/">https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/</a></p>
<p>Debugging one’s app slowdown via metrics as recorded by NewRelic and Honeycomb (with a bit of a plug for Honeycomb’s native support for high dimensional tag values):</p>
<p><a href="https://hackernoon.com/a-short-example-of-why-dimensions-are-suuuuuper-valuable-67e880055eb0">https://hackernoon.com/a-short-example-of-why-dimensions-are-suuuuuper-valuable-67e880055eb0</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Spectre and Meltdown have made the news so publicly that there’s hardly a need to highlight them here. I will mention this article for its good coverage of the industry-standard process for vulnerability disclosure, and how it worked out differently with these bugs:</p>
<p><a href="https://www.theverge.com/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux">https://www.theverge.com/2018/1/11/16878670/meltdown-spectre-disclosure-embargo-google-microsoft-linux</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraDebugging skills are hard won, and hard to teach. Here are some debugging stories to learn from.Tester’s Digest #42: Mutation Testing2017-12-17T00:00:00+00:002017-12-17T00:00:00+00:00http://testersdigest.mehras.net/2017/12/17/testers-digest-42-mutation-testing<h1 id="testers-digest">TESTER’S DIGEST</h1>
<p>ISSUE #42 - December 17, 2017</p>
<p>You use tests to make your code better. You use mutation testing to make your tests better.</p>
<h1 id="topic-mutation-testing">Topic: Mutation Testing</h1>
<p>What is mutation testing: the basic idea is to inject bugs into your source code to establish whether your unit tests would find them, with the purpose of then adding the missing tests.</p>
<p><a href="https://blog.codecentric.de/en/2016/01/mutation-testing-watching-watchmen/">https://blog.codecentric.de/en/2016/01/mutation-testing-watching-watchmen/</a></p>
<p>How to pick the right mutants to destroy in your code by adding only the useful missing tests:</p>
<p><a href="https://blog.codecentric.de/en/2016/02/sensible-mutation-testing-dont-go-killing-spree-2/">https://blog.codecentric.de/en/2016/02/sensible-mutation-testing-dont-go-killing-spree-2/</a></p>
<p>Neither 100% coverage, nor 100% mutation coverage are a silver bullet against bugs. However, mutation testing clearly shows out pieces of code which need refactoring, and causes you to write more asserts and construct more detailed tests.</p>
<p><a href="http://atodorov.org/blog/2016/12/27/mutation-testing-vs-coverage/">http://atodorov.org/blog/2016/12/27/mutation-testing-vs-coverage/</a></p>
<p>How mutation testing fits with agile process: use pairing, apply MT pragmatically to avoid shift from the useful mentality of “Did you think of this corner case?” to the much less useful “These 3% of expressions/mutants are not covered”, and use MT as a tool to promote refactoring in the Red-Green-Refactor cycle of TDD.</p>
<p><a href="https://www.sep.com/sep-blog/2015/07/14/mutation-testing-totally-a-thing/">https://www.sep.com/sep-blog/2015/07/14/mutation-testing-totally-a-thing/</a></p>
<p>A neat story of debugging a Rails issue, found via mutation testing:</p>
<p><a href="https://blog.arkency.com/constructor-for-a-included-module-in-ruby/">https://blog.arkency.com/constructor-for-a-included-module-in-ruby/</a></p>
<p>This is the list of current MT tools I’m aware of.</p>
<p>Mutation testing in Java with PIT:</p>
<p><a href="http://pitest.org/">http://pitest.org/</a></p>
<p>Mutation testing in Python with Cosmic Ray or Mutmut:</p>
<p><a href="http://cosmic-ray.readthedocs.io/en/latest/">http://cosmic-ray.readthedocs.io/en/latest/</a></p>
<p><a href="https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78">https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78</a></p>
<p>Mutation testing in Javascript with Stryker:</p>
<p><a href="https://stryker-mutator.github.io/">https://stryker-mutator.github.io/</a></p>
<p>Mutation testing in Ruby with Mutant or Mutest:</p>
<p><a href="http://blog.arkency.com/2015/06/how-good-are-your-ruby-tests-testing-your-tests-with-mutant/">http://blog.arkency.com/2015/06/how-good-are-your-ruby-tests-testing-your-tests-with-mutant/</a></p>
<p><a href="https://blog.cognitohq.com/how-to-write-better-code-using-mutation-testing/">https://blog.cognitohq.com/how-to-write-better-code-using-mutation-testing/</a></p>
<p>How Mutant gem works in Ruby, in much detail, for those interested in the technical underpinnings:</p>
<p><a href="https://troessner.svbtle.com/kill-all-the-mutants-a-deep-dive-into-mutation-testing-and-how-the-mutant-gem-works">https://troessner.svbtle.com/kill-all-the-mutants-a-deep-dive-into-mutation-testing-and-how-the-mutant-gem-works</a></p>
<h1 id="off-topic">Off-Topic</h1>
<p>Worth learning: what happens when you press “play” on Netflix, from “what’s a CDN” to how machine learning fits in.</p>
<p><a href="http://highscalability.com/blog/2017/12/11/netflix-what-happens-when-you-press-play.html">http://highscalability.com/blog/2017/12/11/netflix-what-happens-when-you-press-play.html</a></p>
<hr />
<p>If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at <a href="http://testersdigest.mehras.net">http://testersdigest.mehras.net</a></p>
<p>If you come across content worth sharing, please send me a link at
<a href="mailto:testersdigest@mehras.net">testersdigest@mehras.net</a></p>
<hr />dmehraYou use tests to make your code better. You use mutation testing to make your tests better.