Five Testers From VChttp://blogs.msdn.com/b/fivetestersfromvc/en-USTelligent Evolution Platform Developer Build (Build: 5.6.50428.7875)VC IDE Testing Tactics (and a beta1 plug!)http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/07/29/201223.aspxFri, 30 Jul 2004 03:30:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:201223MSDNArchive0http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=201223http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/07/29/201223.aspx#comments<P>[Rob]</P>
<P>You've probably had enough time to read the last post...Ok, so maybe we were reallllly busy getting our beta product release squared away so you can all get your free Visual Studio 2005 Express Beta1 downloads!&nbsp; Grab it, have fun, win an <?xml:namespace prefix = st1 ns = "urn:schemas-microsoft-com:office:smarttags" /><st1:mswterms w:st="on">Xbox</st1:mswterms>, tell me about what was good and what needs more! (There are also versions focusing on other programming languages and even web development!). <A href="http://lab.msdn.microsoft.com/express/default.aspx">http://lab.msdn.microsoft.com/express/default.aspx</A><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></P>
<P>***</P>
<P><EM>Tactical thoughts</EM></P>
<P>Thinking about all the things that go into&nbsp;testing feature...is the API working? Are there any attack paths through this feature (is it secure)? Does it integrate with existing features? Does it depend on features (our group or another?!) that might change before lock down? Is the UI compliant with suite requirements?&nbsp; Is the UI clear?&nbsp; Are UI redundancies appropriate? What will our users think about our answers to these questions?&nbsp; What happens to the UI when the product is localized (translated to another language)? What happens to the behavior of the feature when the product is localized? (Add a number of other questions from networking to user permission levels depending on the feature category...).<o:p></o:p></P>
<P>The list appears endless, but luckily we have limited time (some&nbsp;sarcasm on that&nbsp;use of 'luckily', if you didn't notice)&nbsp;to do our testing so we work with a finite set that makes sense for the feature in question.<o:p></o:p></P>
<P>The highest urgency task is to get the feature tested for main stream functionality.&nbsp; What will our customers need this to do?&nbsp; What will our customers want this to do?&nbsp; What assumptions may be made that could get this feature into trouble (and put the user into a confused or frustrated state)?<o:p></o:p></P>
<P>We also want to see that the feature discoverable and useable.&nbsp; The UI needs to indicate what will happen and let users know about progress if this takes time.&nbsp; Keeping UI consistent so different tasks at least 'feel' familiar in the welcoming sense is valuable...but difficult when taking legacy features into a new arena (a challenge for many still useful feature sets now housed in a building of <st1:mswterms w:st="on">.NET</st1:mswterms> splendor).&nbsp; Is there a point to upgrading old UI to match the new style if the old features are stable as is (and use the resources instead for new features and to fix bugs)?&nbsp; Is old UI style a bug or an acceptable artifact of evolution?<o:p></o:p></P>
<P>These questions are what test cases are all about.&nbsp; Each test case seeks an answer. Customer feedback, experience,&nbsp;and research help us to ask the right questions for our current product.&nbsp; Once we have a base set of questions (Test Case Planning is a cyclical process, just as implementation is) we can move forward to measure the status of product drops based on the answers we get on test passes.&nbsp;&nbsp; We build our automation library which allows us to gather answers faster and feel more secure against regressions that could be introduced every time developers touch the product.&nbsp; At the end of the cycle, we have an automation set that allows us to have a level of confidence that sustained engineering will be able to do service pack work while maintaining a high level of regression test coverage (while the front line team moves to the next generation product).<o:p></o:p></P>
<P>[This posting is provided "AS IS" with no warranties, and confers no rights.&nbsp;Use of any included script samples are subject to the terms specified at <SPAN style="FONT-FAMILY: Arial"><A title=http://www.microsoft.com/info/cpyright.htm href="http://www.microsoft.com/info/cpyright.htm" target=_blank><SPAN style="COLOR: #002c99; FONT-FAMILY: 'Times New Roman'"><SPAN title=http://www.microsoft.com/info/cpyright.htm>http://www.microsoft.com/info/cpyright.htm</SPAN></SPAN></A></SPAN>.]<o:p></o:p></P>
<P>&nbsp;</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=201223" width="1" height="1">VC IDE TestingChannel 9 Visits VChttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/06/10/152869.aspxThu, 10 Jun 2004 21:51:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:152869MSDNArchive2http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=152869http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/06/10/152869.aspx#comments<P><STRONG><FONT face=Verdana size=2>[Ron]</FONT></STRONG></P>
<P><FONT face=Verdana size=2>As I type this, some of the <A href="http://channel9.msdn.com/">Channel9</A> dudes are interviewing one of the compiler PM's (Kang Su Gatlin) right outside my office.&nbsp; I wonder how long the lead time is before the video makes it onto the Channel9 site...</FONT></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=152869" width="1" height="1">well, since you asked twice...http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/05/27/143455.aspxFri, 28 May 2004 03:05:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:143455MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=143455http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/05/27/143455.aspx#comments<P>[Rob]</P>
<P>Woon asked in his comment about the % automation we have with our IDE tests.</P>
<P>Currently, we are at roughly 50% automated in the IDE compared to the total number of active test cases.&nbsp; These tests range from DTE manipulations to actual UI interations the way the user would (drag and drop, button clicking, resizing, ...).&nbsp; One way to get at the UI elements is through MSAA.&nbsp; This works fairly well for standard controls.&nbsp; Custom controls make the story more difficult (which is why we resourt to DTE calls if our test harness can't handle certain controls).</P>
<P>It takes about 2-3 weeks for 15 people to cover all the manual test cases we currently have for the VC IDE.&nbsp; And this doesn't include common shell features that the VS team handles.&nbsp; </P>
<P>Our automation's primary purpose is to give us rapid general status on build drops each day (although our automation is split up between daily and weekly test sets).&nbsp; The automation set does find bugs, but by far the most bugs are found through manual testing of new features.&nbsp; Automation is our 'security blanket'&nbsp;&nbsp;as we can get a breadth of coverage in 24 hrs with a second depth after ~72 hours (looking for regressions from the previous known state) rather than waiting for three weeks (which also uses up the whole team's attention) with manual.&nbsp; Both are vital.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=143455" width="1" height="1">VC IDE TestingAnother new VC Tester Webloghttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/05/11/129996.aspxTue, 11 May 2004 22:30:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:129996MSDNArchive0http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=129996http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/05/11/129996.aspx#comments<P>[Ron]</P>
<P><A href="http://blogs.msdn.com/ericflee">Eric Fleegal</A>, a tester on the compiler back-end test team, has joined the weblogging movement.&nbsp; His area of interest and expertise is floating point code generation.&nbsp; (He authored the recent MSDN <A href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/floapoint.asp">whitepaper</A> on the new Whidbey floating point code generation model).&nbsp; I guess it's time to update our VC bloggers list on the left...</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=129996" width="1" height="1">Cool thing I learned about the IDE today.http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/05/06/127593.aspxFri, 07 May 2004 03:42:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:127593MSDNArchive2http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=127593http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/05/06/127593.aspx#comments<P>[Michael] The VS IDE has a ton of cool UI features - many that I'd never ever head of before.</P>
<P>If you're a command line kind guy, you can get some&nbsp;use out of the &#8220;Find combobox&#8221;. I'm sure this feature is in Visual Studio 7.1 +&nbsp;but I&nbsp;also think it's in Visual Studio 7.0.</P>
<P>If you hit CTRL+D from the editor (and I think most other places in the shell) it will take&nbsp;you to the find combo, this combo box right next to the binoculars&nbsp;sitting in an open folder. From there you can type text you want to find, and then hit F3 to search for it.</P>
<P>What's equally as cool is that you can use the &#8220;&gt;&#8221; operator to put yourself in command mode, and from there you get an autocompleting list of all the commands you can&nbsp;issue&nbsp;to the IDE's shell directly.</P>
<P>For example, I placed the &#8220;&gt;&#8221; character in that edit box, then typed &#8220;F&#8221; and it dropped down with a list starting with &#8220;File.AddExistingProject&#8221;. With autocomplete, I got it to type &#8220;File.OpenFile d:\admin.txt&#8221;</P>
<P>I was personally impressed that we support autocompletion from the File.OpenFile command. </P>
<P>There are a ton of other commands available, some of which you might find easier to uncover this way.</P>
<P>It's another way to get around the IDE without having to mouse around, especially if you don't remember all the keyboard shortcuts.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=127593" width="1" height="1">Boing... Boing... Boing...http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/30/124296.aspxSat, 01 May 2004 03:32:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:124296MSDNArchive0http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=124296http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/30/124296.aspx#comments<P><STRONG><FONT face=Verdana size=2>[Ron]</FONT></STRONG></P>
<P><FONT face=Verdana size=2>I noticed Shaykat (over in the C#/debugger team) </FONT><A href="http://blogs.msdn.com/shaykatc/archive/2004/04/16/114899.aspx"><FONT face=Verdana size=2>posted about bug bounces</FONT></A><FONT face=Verdana size=2>.&nbsp; For our test team., these come in threes.</FONT></P>
<P><FONT face=Verdana size=2>The first one we encounter is ZBB - zero bug bounce.&nbsp; This is driving active product bugs down to zero.&nbsp; This is the bug bounce that Shaykat is talking about in his post.&nbsp; Test is involved in this bounce becuase sometimes we have active bugs assigned to us.&nbsp; These could be bugs where the dev is asking for more information or a repro case or, in some minor cases, it could be because test owns development of some product areas.</FONT></P>
<P><FONT face=Verdana size=2>The second bounce is ZRBB - zero resolved bug bounce.&nbsp; This is driving the resolved product bugs down to zero (i.e., verifying and closing them).&nbsp; Bugs in this category are important because, until the fix is verified, we can't be totally sure it's fixed or hasn't caused some other regression.&nbsp;&nbsp;Testing tends to be the most heavily involved in this bounce since (ideally) they have opened most of the bugs and would be the ones to verify the fixes.&nbsp; They are also involved in bugs that weren't opened by testing since those bugs should cause us to ask why we didn't find those bugs and add to our testing if that indicated a test hole.</FONT></P>
<P><FONT face=Verdana size=2>The third and final bounce is ZTBB - zero test bug bounce.&nbsp; This is driving the active bugs against our test automation down to zero.&nbsp; This is important since a test that isn't working can't find bugs or verify the funtionality of the product.</FONT></P>
<P><FONT face=Verdana size=2>This last bounce doesn't get as much attention as the other two; it's generally only an effort&nbsp;for the test team but we do treat our test automation as a real development project.&nbsp;&nbsp; That means our test automation code&nbsp;is checked into a source code control system, bugs against our automation go into an issue tracking database, and we do periodically drive our bug count down to make sure we're not missing issues or test coverage.</FONT></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=124296" width="1" height="1">Prolific, We're Nothttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/29/123441.aspxFri, 30 Apr 2004 03:19:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:123441MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=123441http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/29/123441.aspx#comments<P>[Ron]</P>
<P>Geez the five testers haven't really been posting much lately.&nbsp; I just asked Michael about this and he said that he feels there is a lot of pressure to be absolutely correct in what we say.&nbsp; My take: we've been really busy lately.&nbsp; It's interesting, there's probably different classes of webloggers out there.&nbsp; The folks like <A href="http://www.scripting.com/">Dave Winer</A> and <A href="http://radio.weblogs.com/0001011/">Scoble</A> seem to post with the same frequency as they breathe.&nbsp; We haven't been doing that.</P>
<P>Maybe Michael is right and we feel as though we need every post to be a masterpiece.&nbsp; I certainly have been thinking about a post I've promised on compiler correctness tests a long time ago (but I still haven't posted anything).&nbsp; Maybe we should not worry so much about this and just post what's happening or on our minds more frequently.&nbsp; I guess this post is a beginning attempt to do that.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=123441" width="1" height="1">We Got Slashdottedhttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/19/116064.aspxMon, 19 Apr 2004 20:06:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:116064MSDNArchive0http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=116064http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/19/116064.aspx#comments<P><FONT face=Verdana size=2>[Ron]</FONT></P>
<P><A href="http://developers.slashdot.org/article.pl?sid=04/04/18/0146234"><FONT face=Verdana size=2>http://developers.slashdot.org/article.pl?sid=04/04/18/0146234</FONT></A></P>
<P><FONT face=Verdana size=2>This is about the VC++ 2003 command line toolkit:</FONT></P>
<P><A href="http://msdn.microsoft.com/visualc/vctoolkit2003/"><FONT face=Verdana size=2>http://msdn.microsoft.com/visualc/vctoolkit2003/</FONT></A></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=116064" width="1" height="1">VC IDE Testing: cycle check pointshttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/15/114289.aspxFri, 16 Apr 2004 04:14:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:114289MSDNArchive2http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=114289http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/04/15/114289.aspx#comments<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">[Rob]&nbsp; </SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT><B><FONT face=Arial size=2><SPAN style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Arial">General IDE testing cycle status points (iterations on/between any two points&#8230;)</SPAN></FONT></B></P>
<OL style="MARGIN-TOP: 0in" type=1>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Specification review</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Test Plan Design</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Test Plan Review</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Developer Implementation design</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Developer Implementation design review</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Test case design</SPAN></FONT>&nbsp;<SPAN class=734340501-16042004>[exploratory, user feedback, regression]</SPAN>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Automation Design</SPAN></FONT>
<LI class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Automation Implementation and review</SPAN>
<LI class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Manual</SPAN><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> Test Pass</SPAN></FONT><FONT face="Times New Roman" size=3> </FONT></SPAN>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Automation run</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Project status evaluation and reporting</SPAN></FONT> </LI></OL>
<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Since the organization is broken up between disciplines (Program management, Development, and Test), cross discipline communication helps deal with the implied dependencies in the eleven points above.</SPAN></FONT></P>
<P class=MsoNormal><B><FONT face=Arial size=2><SPAN style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Arial">A typical cycle contains zero or more of the following obstacles based on the above points:</SPAN></FONT></B></P>
<OL style="MARGIN-TOP: 0in" type=1>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">PM (program manager group) delivers spec(s) late, preventing review by schedule</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">test plans delayed by changing specs/parallel product cycle effort lowering priority</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">test plan reviews are cursory, resulting in missed holes fund later which cost more to correct</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">the design is the code</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">well, we can review the code&#8230;or just use the feature&#8230;</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">test cases ended up being designed without complete specifications&#8230;requires revisions regularly at added cost</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">automation gets pushed back due to UI/feature churn making tests obsolete an hour before they are checked in</SPAN></FONT>
<LI class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Automation harness issues; changing requirements; dogfooding delays (a good thing, but causes progress trade-off) [dogfooding:&nbsp; using VC drops to develop automation for testing VC drops&#8230;]</SPAN>
<LI class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Manual Test Pass: can always count on them, but they take too long if automation not strategic or complete (temptation is to delay automation to get complete manual coverage creating an ever deepening hole&#8230;bite the bullet and do the automation right so manual is only what can&#8217;t/shouldn&#8217;t be automated).</SPAN>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">matrix explosion (platforms * SKU&#8217;s * architectures * permission levels;&nbsp; pair wise testing save us!)</SPAN></FONT>
<LI class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">What do you mean, &#8220;make it green&#8221;?!&nbsp; [balancing quality driven vs. date driven product requirements]</SPAN></FONT> </LI></OL>
<P class=MsoNormal><B><FONT face=Arial size=2><SPAN style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Arial">Some random elaborations:</SPAN></FONT></B></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">There&#8217;s a lot of overlap at the beginning/end of product cycles.&nbsp; This really prevents any one discipline from getting a &#8216;fresh start&#8217;, as wrap up work often takes priority over new work (reverse that for PM).&nbsp; Thus, the new work gets delayed quite often.</SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">A note on &#8216;priority&#8217;:&nbsp; Everything is &#8216;pri 1&#8217;, because if it wasn&#8217;t, it was cut.&nbsp; This is because there isn&#8217;t enough time to do everything that &#8216;has to happen before we ship&#8217;.&nbsp; So, &#8216;priority&#8217; has no meaning.&nbsp; Just ignore it.&nbsp; Treat it as someone saying, &#8216;um&#8230;&#8217; or &#8216;er&#8230;&#8217;.&nbsp; I&#8217;m making it a priority to remove that word from tester vocabulary&#8230;.</SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">At one point, our team had fallen significantly behind our goals for automation development.&nbsp; We found ourselves in a position where we needed to keep testing up on implementation churn by providing immediate testing of any dev patch/check in.&nbsp; This constant impromptu testing continuously delayed automation development between formal test passes.&nbsp; We are in a much better state at this time because we followed through on the difficult decision (made with the dev org) to forego the interrupt driven manual testing completely for several milestones and focus on automation (formal test passes still occurred, of course, to cover all test cases for each milestone).</SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">One angle which has been used with automating IDE features can be summed up with a call to &#8220;do everything as the user would&#8221;.&nbsp; Reality set in as conflicts with shared UI continuously broke tests over multiple groups.&nbsp; Since feature innovation can&#8217;t be stalled just to maintain an automation test bed, it became necessary to follow an academic rule:&nbsp; &#8220;a test should test only what it tests&#8221;.&nbsp; In other words, if you aren&#8217;t specifically testing the UI, if at all possible, the test should not fail because of a UI change.&nbsp; Praise be the DTE!&nbsp; (<A title=http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp href="http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp">http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp</A>).&nbsp; For visual studio, feature area automation authors can often bypass UI that they don&#8217;t need to directly test and get at what does need to be addressed with the current test.&nbsp; This has made automation faster and more reliable for our current test harness.</SPAN></FONT></P>
<P class=MsoNormal><FONT size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><SPAN style="FONT-SIZE: 12pt"><FONT face="Times New Roman">[This posting is provided "AS IS" with no warranties, and confers no rights.&nbsp;Use of any included script samples are subject to the terms specified at </FONT><A title=http://www.microsoft.com/info/cpyright.htm href="http://www.microsoft.com/info/cpyright.htm" target=_blank><FONT title=http://www.microsoft.com/info/cpyright.htm face="Times New Roman" color=#002c99><SPAN title=http://www.microsoft.com/info/cpyright.htm style="COLOR: #002c99">http://www.microsoft.com/info/cpyright.htm</SPAN></FONT></A><FONT face="Times New Roman">.]</FONT></SPAN></SPAN></FONT></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=114289" width="1" height="1">VC IDE TestingWhat kind of people are testing this stuff anyhow?http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/03/25/96488.aspxFri, 26 Mar 2004 03:58:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:96488MSDNArchive3http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=96488http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/03/25/96488.aspx#comments<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">[Rob] I know I said the next installment would be on tactics&#8230;but I think in response to feedback<SPAN style="COLOR: navy"> </SPAN>Woon Kiat I&#8217;ll go with &#8216;resources' first.<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Before that, very quickly, there was a comment from </SPAN><B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Verdana"><A href="http://24.odessa.ua/" target=_blank>AT</A> </SPAN></B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">on alpha/beta users submitting automation test cases along with bugs.&nbsp; Generally speaking, repro steps for bugs are added to the regression test case set and automated where appropriate.&nbsp;&nbsp;Directly accepting scripts, however, would cause security concerns.&nbsp; It would be a nice world though to just be able to build a customer &#8216;drop acceptance&#8217; automation test case database&#8230;&nbsp; </SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Wingdings">J</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">&nbsp; Now you have me thinking about a tool to allow steps to be entered through a web page that would build a safe script to run against the product...hmmmm&#8230;.I could build that using the VC++ IDE once I figure out the design&#8230;<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Resources for Test<SPAN style="COLOR: navy"> </SPAN>in VC</SPAN></B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Test candidates right out of school rarely have formal test training.&nbsp; The experience level is also limited to personal code and maybe a few small group projects. &nbsp;This gives us the challenge of identifying those candidates that have a rough talent for &#8216;finding bugs&#8217; or &#8216;breaking things&#8217; </SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Wingdings">J</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> along with a way to navigate through a product and identify where priority bugs may hide.&nbsp; We also look for people who can adapt quickly and apply techniques &#8216;just learned&#8217; to new problem areas.<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">We write test automation and our product is a development tool.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Therefore we also look for people who have development proficiency with C++ (knowledge, style, problem solving, &#8230; pretty much our customer base).<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Our company has training courses (all disciplines) that supplement on the job experience.&nbsp; Everyone on the team is a mentor to some degree or another for every new hire we get. &nbsp;Formal and informal mentors are encouraged in addition to an individual&#8217;s team lead.&nbsp; Acquiring a mentor from outside the team is one way we continue to mix ideas, technologies, and techniques throughout the company.&nbsp; We also like to occasionally reorganize teams to allow people to move to new feature areas and &#8216;stay fresh&#8217; by providing new challenges.<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Product specific training is hands on.&nbsp; For Visual C++ we focus heavily on dogfooding (use the current drops for real world coding of team tools and personal hobby projects) the product.&nbsp; Plenty of texts exist on previous versions of the product to help people understand where we&#8217;ve been. &nbsp;Specification documents (or, when not available, talks with program managers and feature developers) bring us up to date for what to expect in the current version.&nbsp; When one is new to a feature area, reviewing the old test case base and the bug database helps bring you up to date on the current status and where testing has gone before.&nbsp; Particularly interesting are those bugs found by customers, as they show where customers went with the product that the previous test plans did not anticipate (allowing us to learn better what to give priority to in the limited time we have to test).<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Some books found on my office shelf today (more at home where I have a bigger shelf<SPAN style="COLOR: navy">&#8230;</SPAN>oh, and I&#8217;m not saying these are all &#8216;fantastic&#8217; books, though some are, I&#8217;m just giving a snapshot of what&#8217;s behind me): &nbsp;&nbsp;<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Various language books (C++, VB, C#) <o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Various manuals from internal microsoft training courses<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Various manuals from Test conferences (most recent:&nbsp; PNSQC and SQE)<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Debugging Applications for <?xml:namespace prefix = st1 ns = "urn:schemas-microsoft-com:office:smarttags" /><st1:mswterms w:st="on">.NET</st1:mswterms> and Windows</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Robbins<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Lessons Learned in Software Testing</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Kaner et al<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">How to Break Software</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Whittaker<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Design Patterns</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Gamma et al<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Quality Software Management</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Weinberg<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Writing Secure Code</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Howard et al<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Automation Programmer&#8217;s Reference</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Microsoft Press<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Applied Microsoft <st1:mswterms w:st="on">.NET</st1:mswterms> Framework Programming,</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> Richter<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Programming with Visual C++ <st1:mswterms w:st="on">.NET</st1:mswterms></SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Shepherd et al<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Programming Windows</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Petzold<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt; TEXT-INDENT: 0.5in"><U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Inside COM</SPAN></U><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">, Rogerson<o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p>&nbsp;</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p>;-)</o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p></o:p></SPAN>&nbsp;</P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt">[This posting is provided "AS IS" with no warranties, and confers no rights.&nbsp;Use of any included script samples are subject to the terms specified at <A title=http://www.microsoft.com/info/cpyright.htm href="http://www.microsoft.com/info/cpyright.htm" target=_blank><SPAN style="COLOR: #002c99"><SPAN title=http://www.microsoft.com/info/cpyright.htm>http://www.microsoft.com/info/cpyright.htm</SPAN></SPAN></A>.]<SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p></o:p></SPAN></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=96488" width="1" height="1">VC IDE TestingVC IDE testing (issues and processes)http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/03/18/92408.aspxFri, 19 Mar 2004 04:26:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:92408MSDNArchive6http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=92408http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/03/18/92408.aspx#comments<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">[rob] Since Ron has been entering thoughts and comments on compiler testing, I&#8217;d like to add a perspective on IDE testing from VC.&nbsp; I&#8217;ll leave it in notation format to encourage discussion of points for which readers would like more clarification.</SPAN><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">&nbsp;</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Issues to address through IDE testing:</SPAN><o:p></o:p></P>
<UL style="MARGIN-TOP: 0in" type=disc>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Changing specifications over time (some features are completely ambiguous until devs implement, others have varying levels of specs prior to implementation)</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Library of regression test cases, authors having moved to other teams;&nbsp;QA work is more cumulative than dev work in that regression testing currently takes a larger toll than code base maintenance.</SPAN>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Dependecies (teams depend on us, we depend on them)</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Platform coverage</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">SKU coverage</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Multiple paths through UI</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">CodeModel path around UI</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Localization; Data handling and UI</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">SxS, Compatability</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Performance</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">XP Logo</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">&nbsp;</SPAN><o:p></o:p></LI></UL>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Process:</SPAN><o:p></o:p></P>
<UL style="MARGIN-TOP: 0in" type=disc>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level1 lfo2; tab-stops: list .5in"><B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Theory</SPAN></B><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=circle>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Schedule and cost for the milestone before you begin (sounds like a line from Willy Wonka: [school teacher] &#8220;&#8230;switch our Friday schedule to Monday, which means that the test we take each Friday on what we learned during the week will now take place on Monday before we&#8217;ve learned it.&#8221;)&nbsp; Update schedule as milestone progresses to measure against target dates.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Review feature specifications from PM team, providing feedback (expect updates during cycle).</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Review implementation specification from Dev team, providing feedback (updates expected).</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">QA drafts high level test plan and submit to review by Dev and PM teams; update.</SPAN><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=square>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Reference previous product and similar feature test plans</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Incorporate customer feedback and market expectations for setting priorities</SPAN>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Test team contracts attempt to prevent assumptions from&nbsp;leading to test holes across team boundries</SPAN></LI></UL>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Author test cases according to test plan; update plan as needed.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Author test case automation</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Run automation test passes on coverage matrix</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Run manual test passes on matrix subset</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Log bugs, analyze trends, verify fixes.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Analyze automation code coverage (block level) results from instrumented builds. </SPAN><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=square>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Combine results with test plan holes to target next automation set.</SPAN><o:p></o:p></LI></UL>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">&#8216;Dogfood&#8217; the product</SPAN><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=square>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">(Semi-)stable builds are selected by individual testers to use for developing automation (some data driven, some code).</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">App week:&nbsp; break into teams and develop short term projects (some become team tools).</SPAN><o:p></o:p></LI></UL>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Themed bug hunts:&nbsp; &#8220;Bug Bash&#8221;</SPAN><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=square>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">team targeting of a specific Logo, accessibility, localization,.... categories.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Periodic re-education of feature owners on testing aspects</SPAN><o:p></o:p></LI></UL>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Iterate and ship!</SPAN><o:p></o:p></LI></UL>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level1 lfo2; tab-stops: list .5in"><B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Reality, as enforced by those pesky</SPAN></B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> </SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Wingdings">J</SPAN><B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> market needs and restrictions&#8230;</SPAN></B><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=circle>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Level of clear specifications varies greatly by feature and product cycle.&nbsp; This is not something QA can (historically) count on.&nbsp; Working closely with the PM team allows for the necessary information to be focused on creating effective test plans.&nbsp; Various unexpected needs (from marketing to division to group to feature)<SPAN style="COLOR: navy"> </SPAN>continuously challenge the proposed schedule (you know, that thing we all agreed to before the first feature specification was actually written?).&nbsp; Adapt and overcome; priority trade off.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Implementation plans tend to be the actual code from Dev.&nbsp; Monitoring check-ins and dev check-in test reviews helps to prepare for proper coverage</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Some high level test plans get bogged down from lack of feature specifications and test cases develop <I>ad hoc</I> during exploratory test passes.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Authoring automation competes with running manual test cases.&nbsp; The need to report the current product status, in a sense, delays the ability to report a more complete status more often... (Catch 22, but without the airplanes).</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Automation requires a test harness.&nbsp; </SPAN><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=square>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">A test harness requires development and maintenance.</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level3 lfo2; tab-stops: list 1.5in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">VC has moved from an internal tool to a divisional tool </SPAN><o:p></o:p>
<UL style="MARGIN-TOP: 0in" type=disc>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level4 lfo2; tab-stops: list 2.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Still requires development/maintenance participation</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level4 lfo2; tab-stops: list 2.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Supporting/following the gander isn&#8217;t what the goose used to do</SPAN><o:p></o:p></LI></UL></LI></UL>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Coverage matrix bulges and balloons;&nbsp; OS varieties, SKU variations, Side by side installations, run time compatibility issues, pre-release forks, internal drops, priority customer drops, priority customer quick fixes, alphas, betas, somewhere in there we have the primary branch we plan to ship&#8230;</SPAN><o:p></o:p>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Again and again activities we want to do (Pri 2 and 3 test cases; driving pri 2 and 3 bug fixes; driving pri 2 and 3 features) are restricted by schedule needs (competing in the market).</SPAN>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l1 level2 lfo2; tab-stops: list 1.0in"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">dependencies get out of sync and one team may be blocked on another team&nbsp;</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">&nbsp;</SPAN><o:p></o:p></LI></UL></LI></UL>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Next installment:&nbsp; Tactics...</SPAN></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=92408" width="1" height="1">VC IDE TestingExtreme Programming and Ninja Gaiden...http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/03/11/88315.aspxFri, 12 Mar 2004 05:09:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:88315MSDNArchive0http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=88315http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/03/11/88315.aspx#comments<P>[Michael] Hello everyone.&nbsp;I haven't been very active on the blog lately - we're in the middle of a big test pass. Let's see if I can start a discussion.&nbsp;</P>
<P>How many of you out there are using Extreme programming? How large are your teams? Some of us have been having a discussion&nbsp;about it&nbsp;and we keep coming back to the point that someone has made&nbsp;- that no one knows of any large teams using extreme programming. I hope to hear some good stories about it.</P>
<P>For a small glimpse into my personal life - I've been playing the Xbox game Ninja Gaiden. It's been a long time&nbsp;since I've played a game that was this challenging - yet still made me want to keep playing. It's also probably the best looking game I've ever played on a console. I'm starting to wish I was younger though. Perhaps I should fly one of my ten year old cousins up here. I'm having a hard time reacting fast enough to the bad guys.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=88315" width="1" height="1">Answers to Comment Questionshttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/20/76776.aspxFri, 20 Feb 2004 08:58:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:76776MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=76776http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/20/76776.aspx#comments<P><FONT face=Verdana size=2>[Ron]</FONT></P>
<P><FONT face=Verdana size=2>A couple of questions from Woon Kiat to my <A href="http://blogs.msdn.com/fivetestersfromvc/archive/2004/02/10/70438.aspx">previous post</A> on the different types of testing we do on our C++ compiler:</FONT></P>
<P><FONT face=Verdana size=2><EM>&gt;&gt;Is understanding the inner working of a compiler vital to compiler testing?</EM></FONT></P>
<P><FONT face=Verdana size=2>I would say so, at least at some level.&nbsp; In a general sense, I would say it's hard to be really successful at testing anything without understanding the domain area of what you are testing.</FONT></P>
<P><FONT face=Verdana size=2>That said, I don't think many folks came to our compiler test teams with a lot of understanding of the inner workings of a compiler, especially the back end.&nbsp; This is something that we pick up from doing the work.</FONT></P>
<P><FONT face=Verdana size=2><EM>&gt;&gt;How would you verify the binary generated from the compiler is correct? Is it fall under Correctness Tests?</EM></FONT></P>
<P><FONT face=Verdana size=2>In general, we write programs to expose the behavior that demonstrate the particular compiler behavior we expect to see from the feature we are testing.&nbsp; This does fall into the category of correctness tests.&nbsp; I'll try to include some examples of how we do this in my next post on C++ compiler testing.</FONT></P>
<P><FONT face=Verdana size=2>Thanks for the questions, Woon!</FONT></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=76776" width="1" height="1">New Whitepaper: Floating Point Code Generation in Whidbeyhttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/19/76588.aspxFri, 20 Feb 2004 02:22:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:76588MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=76588http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/19/76588.aspx#comments<P>[Ron]</P>
<P>One of the major new features in the area of code generation for the upcoming Whidbey release of VC++ is our new floating point code generation model.&nbsp; One of the members of my team, Eric Fleegal, has written a nice white paper about this feature for MSDN:</P>
<P><A href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/floapoint.asp">http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/floapoint.asp</A></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=76588" width="1" height="1">Why I'm blogging...http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/12/72216.aspxFri, 13 Feb 2004 03:40:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:72216MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=72216http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/12/72216.aspx#comments<P>[Michael] I've got a circuitious style - so if you're impatient, you may want to skip this thread. I'm going to talk about one of the big reasons I'm participating in this blog, and what I'm hoping to gain from it. I'm also big on analogies and stories - and as a fan or Robert Fulghum, I hope this one all makes sense in the end.</P>
<P>I went to a physical therapist a while ago - had some pain in my shoulders and arm. I was in a sour mood. Wasn't as productive at work as I wanted to be. I got there and we talked. I complained a lot about my shoulder and arms. Then a funny thing happened. The PT felt around for the tension, listened to what I had to say about that, and we started working on my lower back. After a while, the focus shifted onto my shoulders and arm and neck. However, most of the time was spent on my back. It wasn't until I was there that I realized that my pain was really in my back.</P>
<P>My back had hurt so long that it had stopped registering with my mind that it was a problem. My neck and arms were compensating for the back pain. I'm glad that the PT didn't just work on my neck and arms. If that had been the case, I'd be right back where I started from. But if the PT has just done what I said without establishing a real dialogue with me, we wouldn't have solved my real problem.</P>
<P>There is something different about the information you get from a simple statement and the information you get by "being there" or "being in touch". There is a lot of context missing from one off communication.</P>
<P>This is one of the reasons I hope this blogging experiment will work.</P>
<P>We have charts and graphs and concerns and suggestions filling up databases. We often have more information than we could practically sort through. And we try. But it doesn't tell the whole story. Without the right context, we might not make as good a decision as we could have with the context customers can provide - if there is real dialogue.</P>
<P>Hopefully, by being out here we can establish a useful and meaningful dialog. Blogs seem to be a compromise in a way - they can't scale if everyone who could read one asked a personal question - but hopefully it is a step toward establishing a strong dialog that will provide us and you, our readers, the context to understand each other. And hopefully solve some problems and relieve some customer "pain". <BR>That's what my opinion of the job of Visual Studio is. To make it easier for you guys to write your applications.</P>
<P>Dialog works both ways too. If you don't know what we're doing, or why, then you probably won't be very eager to buy our product. You might feel like we aren't paying attention to your needs. I hope, I really do, that we're spending more of our time working on your problems than your symptoms. Please let us know if that's not the case.</P>
<P>&nbsp;</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=72216" width="1" height="1">Testing the C++ Compiler – a Three-Pronged Approachhttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/10/70438.aspxTue, 10 Feb 2004 08:53:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:70438MSDNArchive8http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=70438http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/02/10/70438.aspx#comments<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2><STRONG><EM>[Ron]</EM></STRONG></FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>Moving on from my <A href="http://blogs.msdn.com/fivetestersfromvc/archive/2004/01/31/65433.aspx">previous post</A>, I&#8217;d like to start talking about how we approach testing the C++ compiler.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>I&#8217;ll outline the three main approaches we use and in the next three posts I&#8217;ll drill down into each one in detail.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>A disclaimer: the compiler is tested by three teams inside the VC testing group; the front-end team, the C runtime (CRT) team (a lot of the functionality that the compiler relies on is implemented in the CRT), and the back-end team.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Since I am a member of the back-end team, that perspective will be prevalent here.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2><STRONG>Correctness Tests</STRONG></FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The first way that we test the compiler is with what we call &#8220;correctness tests.&#8221;<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>These are tests that exercise the compiler by giving it code to build and, in most cases, exercise the generated code and verify expected behavior.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Cases where we would not build would include diagnostic tests.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>These are tests that give the compiler code that should not compile and verify that the correct error message is generated.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>Our correctness tests are arranged into a number of suites.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Some are implemented by us and some are implemented by 3<SUP>rd</SUP> parties.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>An example of test suites implemented by 3<SUP>rd</SUP> parties that we use would be some of the commercial 3<SUP>rd</SUP> party C++ conformance suites.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Our correctness tests are implemented using a test harness written in Perl.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>The harness is pretty minimal but very flexible.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>It has served us well for many years now.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>For comparison, the IDE team is on their third test framework since I joined the team.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>The compiler team is still using the same framework they were using when I joined the team.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>There is a benefit to simplicity.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2><STRONG>Real-World Code</STRONG></FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The second way we test the compiler is with what we call &#8220;real-world code.&#8221;<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>What that means is that we use our tools to build large code bases.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>This is in contrast to our correctness tests which are usually very small, focused tests.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>For this form of testing we build some of the larger code bases in the company.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Real world code building has two big benefits: 1) we exercise our tools in large, real-world scenarios that are just not possible with our correctness tests, and 2) we demonstrate to ourselves and our internal customers that we can handle their code bases with our new tools.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>This helps convince those customers to adopt our newer toolsets which is a benefit to both them and us.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2><STRONG>Benchmarks</STRONG></FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The third approach that we use to test the compiler is benchmarks.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>These are scenarios that are used to measure the performance of generated code.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>In my next post, I&#8217;ll drill down into the details of our correctness tests.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>In the mean time, feel free to comment on this post and ask any questions you may have related to testing the C++ compiler.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>I&#8217;ll do my best to answer them.&nbsp; Thanks for reading!</FONT></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=70438" width="1" height="1">C++ Compiler TestingTesting the C++ Compiler - Introductionhttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/31/65433.aspxSat, 31 Jan 2004 08:41:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:65433MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=65433http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/31/65433.aspx#comments<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2><STRONG><EM>[Ron]</EM></STRONG></FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>Inspired by <A href="http://www.jixal.com/blog/archives/000085.html">Gus&#8217; recent post on testing the C# compiler</A>, I thought I&#8217;d talk a little bit about how we test the C++ compiler.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Before I do that, though, I should talk about the organization of the VC test team to put things in context.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>I&#8217;ll do that in this post and focus on the compiler in future posts.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The VC test team (VCQA) is organized into three sub teams: IDE/Box, Programming Model, and Code Generation.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The IDE/Box QA team is responsible for the parts of VC that live in the IDE such as the VC project system, C++ intellisense, and the C++ wizards and designers.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>The &#8220;Box&#8221; part of this group is responsible for the QA parts of release management and other things such as setup, samples, and international aspects of the product.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>In general, box areas are areas of the product that don&#8217;t have developers on the VC team.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Most of us FiveTestersFromVC are on the IDE/Box team.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The Programming Model QA team is responsible for the Visual C++ libraries (CRT, MFC, ATL) and the compiler front end.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN><A href="http://blogs.msdn.com/arich">Andy</A> is on this team.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>The Code Generation QA team is responsible for the compiler back-end, the 64-bit .NET JIT compilers,&nbsp;and some of the tools (primarily the Linker).<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>The Code Generation QA team is subdivided into 4 smaller teams: x86 back-end, IA64 back-end, AMD64 back-end, and 64-bit JIT compilers (the x86 JIT compiler lives on the CLR team).<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>I am the test lead for the x86 back-end team.</FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p><FONT face=Verdana size=2>&nbsp;</FONT></o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Verdana size=2>Next time I&#8217;ll start focusing on how we test the C++ compiler.</FONT></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=65433" width="1" height="1">C++ Compiler TestingLink Maintenancehttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/23/62017.aspxFri, 23 Jan 2004 09:55:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:62017MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=62017http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/23/62017.aspx#comments<P>[Ron] I updated our VC bloggers links to point to their new <A href="http://blogs.msdn.com">http://blogs.msdn.com</A> homes.&nbsp; (I didn't see a new address for Martyn.&nbsp; Did he stop blogging?)&nbsp; I added a link to Andy Rich's weblog.&nbsp; Andy's a tester working on the compiler front-end team.&nbsp; He's actually turned out to be the most prolific of the VC++ bloggers.</P>
<P>I also noticed that a couple of debugger developers have started weblogs: Andy Pennell and Steve Steiner.&nbsp; I'm really looking forward to their posts.&nbsp; Their links have been added to our growing list.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=62017" width="1" height="1">you know you are busy when...http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/22/61938.aspxFri, 23 Jan 2004 04:45:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:61938MSDNArchive0http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=61938http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/22/61938.aspx#comments<P>[rob]I've just stepped out of my office to go to another meeting, and I hear my phone ring...the double ring of an external call.&nbsp; I look down at the floor through the open doorway.&nbsp; </P>
<P>&#8220;I am not 'in the office'....&#8221;</P>
<P>I continue to the meeting that I am already late for.</P>
<P>So, I've been pretty quiet on the blog....it's been busy over here!</P>
<P>I have been attending a local users group for .NET developers.&nbsp; I had originally attended thinking it was generic.NET, but have since learned that most attendees are either VB or C# devs.&nbsp; I may very well be the only VC developer regularly sitting in on these meetings.&nbsp; It speaks to the benefit of .NET that&nbsp;it hasn't made much of a differnence (except when I can't help but point out that MFC is not VC.NET to some of the other developers who have no VC.NET experience).</P>
<P>I have found out that my perception is a bit skewed by my exposure to Whidbey.&nbsp; I was debating this week that VC.NET was just as fast as other languages.&nbsp; It turns out that I was not quite correct.&nbsp; In Everett (7.1), .NET from VC is, sadly, slower for some scenarios.&nbsp; For Whidbey, it is actually slightly faster! (but JIT compiling will be the near equalizer for all).&nbsp; I suppose if you average all these statements I was...correct? (or can this kind averaging only be done for election predictions?)</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=61938" width="1" height="1">We're also at http://blogs.msdn.comhttp://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/10/49471.aspxSat, 10 Jan 2004 22:23:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:49471MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=49471http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/10/49471.aspx#comments<P>[Ron]</P>
<P>Just in case there's anyone who doesn't know, we've joined the crowd at <A href="http://blogs.msdn.com/">blogs.msdn.com</A>.&nbsp; So you can now access us at <A href="http://blogs.msdn.com/fivetestersfromvc">http://blogs.msdn.com/fivetestersfromvc</A>.&nbsp; The old url at weblogs.asp.net still works too.</P>
<P>Hope everyone had a good holiday season...</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=49471" width="1" height="1">Back from the Holidays!http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/08/48863.aspxFri, 09 Jan 2004 04:57:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:48863MSDNArchive5http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=48863http://blogs.msdn.com/b/fivetestersfromvc/archive/2004/01/08/48863.aspx#comments<P class=MsoNormal style="MARGIN: 0in 0in 0pt">[Barton]</P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt">Yes, we are alive.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>We were all out of office for the holidays and now we are back and ready to start blogging. We apologize for being such poor bloggers, and plan to better this year (a new years resolution?).<SPAN style="mso-spacerun: yes">&nbsp;&nbsp; </SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p>&nbsp;</o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt">Today we met and were discussing what we should be talking about in our blog.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>Some of the topics we considered were</P>
<UL style="MARGIN-TOP: 0in" type=disc>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">managing large projects</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">finding bugs</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">talking about VC features</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">getting feedback on VC</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">testing methodologies</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">robotics<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>- teaching kids programming</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">what have I learned working at MS for the past 8 years (today is my 8 yr anniversary)</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">testing in an agile development world</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">test first development</LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">bugs to avoid in the product </LI>
<LI class=MsoNormal style="MARGIN: 0in 0in 0pt; mso-list: l0 level1 lfo1; tab-stops: list .5in">prioritizing and making tradeoffs when fixing bugs</LI></UL>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt">Let us know if you are interested in any of these topics or any other topics that you would like to hear about.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><o:p>&nbsp;</o:p></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt">We are investigating the question of sending bugs to MS.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>We found a process that exists but it looks more like a black hole than a mechanism to get feedback to us.<SPAN style="mso-spacerun: yes">&nbsp;&nbsp; </SPAN>As soon as we find the <U>right</U> solution we will let you know.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=48863" width="1" height="1">Hellohttp://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/05/41538.aspxFri, 05 Dec 2003 22:56:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:41538MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=41538http://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/05/41538.aspx#comments<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Verdana">Hi, My name is Barton and I&#8217;m the Test manager for Visual C++.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>The VC Test Organization includes teams focused on the User Interface, Programming Model (C++ Front End and Libraries), and the Compiler Back End (code generators for x86, ia64, and amd64 architectures). <SPAN style="mso-spacerun: yes">&nbsp;&nbsp;</SPAN>I&#8217;ve worked at MS for the past eight years and have always worked on the VC Test Team.<SPAN style="mso-spacerun: yes">&nbsp; </SPAN>I started testing the SQL Debugger and moved up to lead the User Interface Test Team and then over to lead the Compiler Test Teams and am now responsible for all VC Test Teams. <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></SPAN></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Verdana">Thanks, Barton&#8230;<o:p></o:p></SPAN></P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=41538" width="1" height="1">IntroductionsReporting VC Bugshttp://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/04/41375.aspxFri, 05 Dec 2003 05:21:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:41375MSDNArchive2http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=41375http://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/04/41375.aspx#comments<P><EM><STRONG>[Ron]</STRONG></EM> Matthew in his comment to Rob's introduction asks:</P>
<P><EM>I'm curious. I'm sure most people on a daily basis find problems with the IDE (or improvements). But for mere mortal developers it's not transparent as to how to report these (or if they have been reported in the first place). Any suggestions on improving this process?</EM></P>
<P>I think the standard answer to this question is the same as it's been for a while: call product support and report the bug.&nbsp; But I've also seen many complaints about this particular method of reporting bugs since, from what I understand,&nbsp;you have to give all the information needed to open up an incident (including credit card info if you're not using an incident that came with some Microsoft product, like MSDN) and it's up to the support engineer to remove the charge.</P>
<P>I do know that we are pretty good about responding to posts on the microsoft.public.dotnet.languages.vc newsgroup but depending on where we are in the product cycle, YMMV.&nbsp; Sometimes we need to focus on the private beta or alpha newsgroups, for example.</P>
<P>Certainly, if you post a problem here in a comment to one of our posts, we'll make sure to investigate it and take the appropriate action.</P>
<P>(BTW, don't limit your complaints to the IDE only.&nbsp; If you have issues with other areas of VC++ (compiler, etc.) we'd like to hear about those as well.)</P>
<P>I'll bring this up with&nbsp;the powers that be&nbsp;and post back if I get any&nbsp;better information.</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=41375" width="1" height="1">Don't forget me...http://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/04/41365.aspxFri, 05 Dec 2003 04:32:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:41365MSDNArchive2http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=41365http://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/04/41365.aspx#comments[Rui] I am Rui. I am on the C++ project team responsible for the project management and build.&nbsp; If you have any questions, suggestions or grievances, bring them on.&nbsp; Let's see how we can make your project building experience more enjoyable.<div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=41365" width="1" height="1">Introductions[witty_salutation] hi [/witty_salutation]http://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/04/41300.aspxFri, 05 Dec 2003 00:50:00 GMT91d46819-8472-40ad-a661-2c78acb4018c:41300MSDNArchive1http://blogs.msdn.com/b/fivetestersfromvc/rsscomments.aspx?WeblogPostID=41300http://blogs.msdn.com/b/fivetestersfromvc/archive/2003/12/04/41300.aspx#comments<P><STRONG><EM>[Rob] </EM></STRONG>I'm Rob. Being one of two team leads for the VC IDE let's me get my fingers into the VC Wizards, .NET designers for VC, VC debugging experience, and the VC Resource Editors (and generally everything else as well). I've been testing at Microsoft since 1999. Before working in IT, I was doing research in molecular biology (and offended many millions of Tetrahymena thermophilia, I'm sure).</P>
<P>Welcome to our weblog!</P>
<P>-rob</P><div style="clear:both;"></div><img src="http://blogs.msdn.com/aggbug.aspx?PostID=41300" width="1" height="1">Introductions