Let us talk about “Quality Bars”

Let us talk about “Quality Bars”

When we talk about quality bars, we are not talking about the phenomenal gems such as the Kalahari Oasis in the Mabalingwe game reserve, which I visited and thoroughly enjoyed on more than a couple of hot days under the sun.

Instead we are starting a discussion on the quality bars we are using and considering for ALM Ranger projects, with the hope that you (ALM community) will give candid feedback and share your experience.

Why do we need quality bars?

We strive to ship quality solutions and experience which:

Accelerate the adoption of Visual Studio with out-of-band solutions for feature gaps and value-add guidance

Create raving and unblocked fans

Meet regulatory and legal requirements

The first has been our mission from the start and the second will always be fuelling (inspiring) us and the community. The third is the one that has and will continue to introduce challenges and friction in the ALM community, because it requires bandwidth (which we have little of) and razor-sharp focus.

What are the challenges with quality bars?

Let’s start by looking at the quality bar slider diagram we created in an attempt to highlight the challenges when striving for the full compliance with industry guidelines, standards and a quality user experience. At this point we assume that the ALM Rangers are well known to you, as well as their geographically dispersed, virtual, part-time and volunteer based community.

When we talk about tooling we can move the slider from none (pure guidance) to a fully supported “shrink-wrapped” product. The compliance and quality bar checks and balances take a huge jump when we reach and especially cross the “ship binaries” marker.

As mentioned the ALM Rangers community is based on volunteers, who invest their passion for technology, their real-world experience and their personal time on a part-time basis.

What motivates the ALM Rangers? Great question which probably will never have a finite answer. Here are a few replies from a recent survey in which we asked ALM Rangers why they do what they do:

Collaborate with passionate real-world experience

Ability to work on latest technologies and professionals on valuable projects

Credibility in the field and to share real world experience amongst fellow Rangers

To develop solutions for real world impediments and share them with the whole community outside the rangers

Work together with other high passionate people on subjects of my interest

Opportunity to create things valuable to the wider community, opportunity to influence the product

Contributing to others success and provide a value for the community around Visual Studio

Because I enjoy helping

…

The challenge is that the motivation factor starts declining as we move the quality bar slider to the right as shown.

As we move the quality bar slider to the right we get closer to the product group quality bars, but also start inheriting more and more “process tax”. Quality solutions, quality experience and in particular regulatory and legal requirements introduce stringent compliance testing and validation. We refer to this as the “process tax” which requires time, razor-sharp focus and which inadvertently chisels away at the motivation factor for volunteers, who want to build solutions in their personal time and not be bogged down with compliance red-tape.

Our current strategy is for the Program Manager Product Owner and Dev Leads to shield the team from as much of the quality processes as possible, keeping the team focused on building and testing the bits and continuing to have “fun” with what motivates them.

Code Review checks are introduced when the solution contains source code that is more than just sample source code and are typically used by the community to create a binary. Quick Response Sample Tooling are typically shared as source code only and the community creates and shared the binary builds. See http://aka.ms/treasure35 for examples. Effort is estimated on our backlogs in hours.

Quality Essentials checks are used whenever a binary is included with the solution. Effort is estimated on our backlog in hours to days, making it impractical for quick response solutions.

But, what about testing in the context of the ALM Rangers?

Automated Unit Testing … typically includes a mix of unit, system, integration and regression testing that is automated and performed as part of the build.

Manual Testing … typically includes a mix of manual user, exploratory and user acceptance testing to validate features and edge-cases, using Microsoft Test Manager.

Dog-fooding … typically includes a mix of manual, exploratory and on-the-job testing with no predefined tests. While we have dog-fooded the use of automated Coded UI testing, the preferred trend is to use humanoids who are passionate about the technology, who test edge cases we would never have considered and who are willing to give candid feedback.

Your candid feedback and food for thought would really be appreciated! Rangers, comments?