Steps to Open Actionable Bugs

Hi, I am Vaishnavi Sannidhanam, test lead from the Visual C++ Compiler Backend team. I joined Visual C++ team 4 years ago and helped ship VS2005 and VS2008. In this blog, I would like to introduce you to the most effective way of reporting actionable bugs to us.

Where to open the bug

When you find a bug, the best way for you to report it, is to reach us through Microsoft Connect. To report a bug through Microsoft Connect, open the Connect page (https://connect.microsoft.com/) and click on “Your Dashboard”. Select the project the bug pertains to and proceed with opening the bug.

Usage Scenario

Giving a description of why you are doing what you are doing, to help us understand and evaluate the importance of this bug fix for you.

Providing us with a repro

A repro is nothing but a series of steps that reproduce the behavior you are seeing. It is always useful to give us a repro and the smaller a repro the better it is for us. Sometimes you might not be able to attach a repro case through Microsoft Connect because of the size of the repro. If that is the case, please do make a note in the bug that you have a repro that you were not able to attach and we will make other arrangements to get that from you. It would also be great if you provide us with the expected and the actual behaviors. For UI/accessibility/localization related issues, it is very useful if you were to give us a screen shot of the broken behavior. It would also be very valuable if you let us know what version of the product you found the bug in.

Below are specific issues where special pieces of information pertaining to the bug would greatly help us

1.Project conversion issues: It would be tremendously helpful for us when the opened bug contains the following information

1.1.Original project file

1.2.Conversion log (upgradelog*.xml which can be found in the solution or project directory depending on what is getting converted)

1.3.Description of the problem you ran into

2.Visual Studio Crashes: These are usually reported to us as Watson reports hence nothing further needs to be done. However, if for some reason you are not getting a Watson prompt, you can open a bug with the repro steps detailing the version of VS, the type of the project (you are more than welcome to share out the project with us as this improves our chances of reproducing the crash), the series of actions you performed that caused VS to crash, etc. If the crash is on Win7, then the usual JIT window to attach debugger after crash will not be shown, by default the settings hide it. If you want to debug through the issue, here are steps to turn the debugger on:

2.2.Select “Each time a problem occurs, ask me before checking for solutions”

Alternatively if your group policy blocks this setting you can click on Select programs to exclude from reporting and add – %ProgramFiles%Microsoft Visual Studio 10.0Common7IDEdevenv.exe and %ProgramFiles%Microsoft Visual Studio 10.0VCvcpackagesvcpkgsvr.exe

3.Intellisense failures and browsing related issues: You can use the new dev10 feature that enables logging. To enable this feature you can go to Tools à Options à Text Editor à C++ à Advanced à Logging Level and set this to true and level to be 5. By default this is disabled. Once enabled, you can reproduce the problem and attach the resulting log. You can see this log through the output window by selecting “Visual C++ Log”.

4.Memory Consumption Issues: It would be easier for us to track down memory issues if you let us know the amount of memory being used along with a detailed description of what type of actions is causing this to happen. Sharing out your project or a project that exhibits this problem as a repro case would greatly increase our ability to repro this bug on our end. You can also use the sysinternals tool vmmap (http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx) to look at various memory metrics such as virtual memory, working set, native heap, managed heap etc. You can create vmmap files by running vmmap.exe, attaching to devenv.exe when it is in the high-memory state, and selecting Save from the menu. This saves some data about how the memory is being used in the system which allows us to more easily spot where to start looking for the problem. You can attach this file as part of the bug as well.

5.Project Build issues: Detailed build log will be very helpful when you hit project build issues. You can enable detailed logging by going to Tools à Options àProjects and Solutions à Build and Run, set “MSBuild Project build output verbosity” to “Diagnostic” in IDE build or pass /v:diag switch to MSBuild for command line build. It will be helpful if you can also provide the build log from previous versions of Visual Studio.

6.Compile Time Issues: You can produce a pre-processed file or a link repro depending on what kind of problem you are running into. The steps on how to generate these files are documented at http://support.microsoft.com/kb/974229. Also it is always useful to gather all the switches you are using when doing a compile. By default the switches passed to the compiler will not be shown in the output window and you can turn this on by going to Tools à Options àProjects and Solutions à Build and Run, set “MSBuild Project build output verbosity” to “Diagnostic”.

Pre-processed files:

The very best kind of repro is the small section of source that compiles all on its own without any dependencies and produces an executable that reproduces the problem. If you have this kind of a dependence free repro, please report it along with what wrong behavior you observe and what you think the right behavior is. However, it is very hard and time consuming to produce a dependence free repro. That’s why the next best kind of repro is the single, pre-processed file.

For managed code, it is more useful for us if you give us the preprocessed file (following the instructions above) along withall the assemblies referenced during compilation. You can find out what assemblies are referenced by adding /showIncludes to the command line or by clicking no project properties à Configuration Properties à C/C++ à Advanced à Show Includes à Yes

Reporting bugs in the pre-processor:

For bugs in the pre-processor (i.e. compiling something /P or /EP doesn’t yield what you think it should), a more elaborate repro helps us out. The elaborate repro is nothing but the main one source file where you noticed the problem along with all the others that the main one #includes.

The easiest way to obtain all the source files is to build your repro case with /showIncludes. This will list all the files that are brought in during compilation. If you copy all those files to a directory, you should be able to reproduce the problem in isolation from the rest of your sourcebase. For instance, you should be able to set INCLUDE=. and still be able to produce an object file.

Link Repros:

Link repros are used when bugs are occurring during link time for both LTCG and non-LTCG cases.

To generate repros for problems with Profile Guided Optimization (PGO): Problems with the Instrumented Build, i.e. passing /PGI, simply follow the link repro instructions detailed athttp://support.microsoft.com/kb/974229. All needed files will be copied over to the link repro directory.

To generate repros for problems with Profile Guided Optimization (PGO): Problems with the Optimized Build, i.e. passing /PGO, simply follow the link repro instructions detailed at http://support.microsoft.com/kb/974229. In addition, include the .PGC files from your scenario that is causing the problem you want to report.

7.Application Runtime Issues:

Floating point precision issues

After using the above described techniques to reduce your repro size, letting us know what your expected result is and what we are producing is very valuable for us. Also helping us understand what kind of an impact this precision error has on your product would be useful for us when we decide on when to fix the bug. You might also want to look into the various floating point options that the compiler provides that help you choose between precision and performance (http://msdn.microsoft.com/en-us/library/e7s85ffb(VS.100).aspx ).

Application crashes

When your application crashes and you think it is a compiler issue because

you don’t find any fault in your code

you see that this crash happens with the new version after upgrade and does not happen with the old version

you see that the crash happens only in release mode and not in debug mode

In these cases you could provide us with the source code that exhibits this behavior. We would really appreciate it if it were a reduced repro case as this tremendously helps us in narrowing down the problem.

Thank you for reading through this blog. I hope this blog enables you to open actionable bugs which in turn will help us ship a higher quality product. Thank you for dogfooding our product.

@Teo: I think his blog post was more useful (and took more effort) than your offering. How would you feel if you took the time to do a similar post about your work (presuming you work) and then you instantly received a response like yours?

If you find a compiler bug by looking at the code that the compiler generated (by looking at asm) then you can definetely open a bug and give the description of what the compiler is doing wrong along with the wrong asm sequence.

@dg, when you receive a call from a customer who has just lost a sizable chunk of their data (talking about terabytes back in 2003) and you trace the problem to wrongly generated code by the compiler; and then report this to MS with a 7-line code sample reproducing the bug; and then MS closes the bug with "yeah thanks, we will look at this sometimes later", THEN come back and tell me how to write my posts.

Anyway, we are all programmers, let’s test how the system works, shall we?

1. When you talk to people, words like "actionable" (meaning "something we can act upon"), "leverage" ("use") and so on just sound plain stupid. That’s if your audience actually knows English. I do not know it, so they do more harm, instead of giving useful information.

2. I am the guy who wrote part of the rules for our testers about how to report useful bug reports. And their essence was the same as this blog post. I am surprised that it was even *necessary* to write it. I mean, Visual Studio users are programmers, not car mechanics. Programmers should know what information they need to reproduce and fix a bug. So it should be very easy for them to generate useful bug reports. I know, it is not very logical :)

If Vaishnavi Sannidhanam is really serious about this blog post, he/she might use my latest bug report as a test subject, dissect it, and point to us what is good, bad or just ugly about it, so that we can learn from our mistakes and by example. Now, does my offering sound useful to you?

It told the VC team something they need to know: That right now, the feedback we get when reporting bugs is ridiculous. They’re masters in giving the impression that "We don’t have time to fix the bug *today*, so we’re tossing it in the garbage bin *forever*".

From where’d all the nasty boys come? When in Rome, you speak lika dis and lika dat (yuh, with the fingers flick under the chin, ya know). When in an MS blog, you are to understand everything is super good, even those actionable issues that were mistakenly architected when they should have been properly leveraged to … I’m lost. I think that’s actionable. Not sure. Not super sure, anyway. Something in my architecting. Need to leverage better.

I work in the Visual C++ team and just thought I would give some context so maybe it’s a little easier to understand why Vaishnavi made this post. We get a variety of customer reports from all different types of people. We would love to be able to help everyone that has problems but we do have to prioritize and apply resources to the most impactful problems. If you give us a detailed report with specific instructions on how to reproduce the problem including specific hardware that really helps to be able to isolate the problem and come up with a solid fix. There are instances where people say I have a bug with Visual Studio and here is the code I wrote can you help me? As you might imagine this could be quite time consuming and in the end it may just be a bug in the customer’s code.

As to when bugs get fixed – this is impacted by the severity, discoverability, work arounds, and to some extent where our group is in the product cycle.

We want to make our customers satisfied and happy with our work. We try and give as much information to them as we can ease their problems. I am sure that sometimes a reply can seem curt or dismissive. We have a variety of people each with their own style so this may be true in some instances.

We all hope to do better and make your experience using Visual Studio a positive one.

Rick, I don’t think you and many other engineers and managers at Microsoft get it. We’re all engineers, we understand that there are priorities, what upsets many of us is how you deal with bug reports–how you flippantly respond, how you pretend there is no problem, how you repeatedly ask for steps to repeat when the bug is blindingly obvious and then how you close or postpone *bugs*–how can you close or resolve a bug when you haven’t actually fixed it?

To add insult to injury, there are serious bugs in Visual Studio that have been there for a long time. We see a lot of work done on the trivial edges, but these core bugs ignored. It seems to many of us that your priorities are really screwed up–that some marketing manager is more concerned with flash than substance. Again, most of us are engineers and know what this battle is about. When dealing with commercial shrink wrapped product, the marketing managers are often right; moving product requires flash! But here we are dealing with a product intended for engineers who in turn write product that helps sell windows. Visual Studio is not an end in and of itself, but is being treated as it is.

My biggest frustration with VS 2010 is that you have spent so much time on flash and created a slower, sometimes much slower, product while ignoring all the promises you’ve made over the years. In several cases, you’ve just made things worse.

For a time, the motto was "10 is the next 6", but it isn’t. Not even close. VC++ 6.0 had problems (which people often forget when looking back; I use it daily so am reminded of its shortcomings daily) BUT it was a lean, fast product and at the time, there was nothing better. With a few add-ons, 6.0 is still a very viable product–fix the compiler to be more compliant, add tabs and intellisense at least as good as VS 2008, make it work with the latest SDKs and I’d stick with it. (SDK access is ultimately driving us, as a group, to VS 2008; I have add-ons which provide much of the missing functionality and aren’t bothered by having to switch to VS 2005/2008 just to write the .NET components.)

There’s the criticism, now here’s my suggestion: Microsoft needs to strengthen the testing group and make it more independent from engineering. Engineers should be able to read Connect, but not do anything else with it. A tester takes a report and turns it into an internal bug report with reproducable cases (if possible, sometimes it’s not.) If the tester needs more information, they elicit that.

I’d go one step further and make the entire process more open. Again, Visual Studio is a tool to help developers create products which in turn greatly help sell Windows, Office and SQL Server. It is aimed at developers. Make the bug lists transparent. Let everyone know what the documented list of bugs are, which ones aren’t being fixed, which ones are and so forth.

While Microsoft has always been a very closed shop, so to speak, it does seem to me that since the late 90s, soon after VC++ 6.0 was released, that the company has become even more closed and insular, especially with its developer program. It seems to have put marketing before engineering. It seems to have favored style over substance. There is a flippancy from many of the engineers that wasn’t there seven or eight years ago. Above all, it seems that the groups at Microsoft that create developer tools have grown increasingly insulated from what real world developers need. (To illustrate; the point of using C/C++ is to create lean, fast programs, but there are many trends leading away from that. Some groups seem to take this to heart while others are oblivious–the CRT code has gotten better with 2010, but the MFC code has gotten significantly worse.)

In conclusion, you are seeing a backlash from developers who have simply gotten tired of being treated with contempt by the very company they helped build.

I fully understand your point of view and perception of the product AND the team.

I’ll ask you a follow on question: Do you think it’s possible that our intentions are different than how you perceive them? Let me explain.

You talk about Marketing managers and engineers in your post. You also mention that our handling of connect bugs is not as good as it should be – which I agree with. Our testers are engineers after all. And when they try to repro issues, and they can’t, we let that engineer reply to your bug directly – not with a marketing manager that is eloquent and skilled at communication. We try to educate the whole team as to who you should interact with customers. But in the end, not all engineers are great at that part and you may get very poorly worded answers. And then you perceive that as us not caring.

I’ve worked in the Windows org for many years. When I joined the VS team I was impressed at how easy it was for all of you to open a bug that came directly into my inbox – unfiltered by anyone. That has it’s pros and cons. We get all the feedback. But we do get a lot of junk thrown at us we have to sift through, and you get unfiltered answer back.

I’ll reiterate that you bring up lots of good points in your post. If you feel the product is slow and sucks in various areas, I respect your opinion. We have to do a better job.

I would just ask you don’t assume and then assert we don’t care. We try our best to deliver a product you want to use. If we are not doing that, we need to take your feedback about the product and try harder.

I’m not saying you don’t care; I’m talking here about the perception among the developer community and right now it’s about as bad as I’ve seen since the DOS days.

Part of this frustration and distrust is that developers see Microsoft spending a lot of time and effort doing things that don’t help them in the now WHILE ignoring the bugs and issues that do affect them. In other words, we see too many issues being ignored in favor of features.

Then, when we report these issues, we’re too often told that there isn’t time. But why would you have a beta if you intend to punt all the issues, especially the hard ones? And if you ignored an issues with 2005 and then with 2008 and now with 2010, why would anyone believe you won’t do it again in the next version?

RE: Testing

(First, I use the word "tester" to mean the engineers who do testing and "engineer" to mean the programmers who actually develop the software to be tested.)

My view is that Testing and Engineering are two separate and vastly different functions and a fundamentally adversarial one. The job of testing is to prove a product doesn’t work, not to endorse what engineering has done. (In other words, testers should be trying to prove that the engineers are idiots and the engineers should be trying to avoid being proven such.)

The reason I prefer testers to deal directly with Connect, or something comparable, is not only because it’s an obvious extension of their job (to find bugs), but because they don’t have a vested interest in defending the product and/or design choices made. They are, by definition, much more objective.

Another advantage of having testing dealing with "the public" is that they learn what the expectations are versus what is begin delivered. In addition, they can coalesce issues that users may see as being separate, but which are, in fact, the same thing or at least related.

They also learn to formulate responses appropriate to the reporting party. Responding to an issue report from an engineer is very different than responding to an issue report from a casual user. (And I, for one, am not bothered being treated bluntly–I prefer it over the politically correct kumbaya crap–but and bothered when I’m told that something which is clearly a bug is "by design", or slightly less awful, that it won’t be fixed, whereupon it is declared resolved and closed.)

Don’t get me wrong, I do like it that Microsoft Engineers see the raw bugs; that is very good. Direct feedback is invaluable. I’m also quite sure that sometimes those engineers know the fix and want to do the fix, but are prevented from doing so by the Pointy-Haired Boss.

(Which does raise the issue if whether the real problem isn’t outside developer dealing with inside developer, but whether many of those bugs, suggestions and concerns are being escalated to management and the architects. Yeah, speaking rhetorically.)

VC++6 IDE was and is the best VC++ IDE and all of the new versions after VC++6 is just getting worse except for 2008 which is just a little better than 2005 and a lot worse than 6.

You know why? That’s all because of .Net crap.

MS is continuing it’s wrong way by using WPF as vs code editor.

More .Net = More Bloat = Less Performance

Today I punched my office computer very hard all because of that god damn it piece of crap VS2005 SP1 is crashing every 15 minutes. It says there is not enough memory. I don’t know so you tell me, isn’t 4GB of ram enough to run a program designed for 2005 year computers?

It helps to not re-coin well known computer terms with new ones. Creating new terms, or in this case, borrowing legal terms (actionable) out of context, only serves to confuse. Scrum/Agile are good examples of the large amount of extra confusion one can add by making up terms for existing well know concepts.

I filed a bug which was closed as notabug. I believe the person who closed the bug was wrong, but I have no way to reactivate the bug or anything like that. Such experience *sucks*. You got to work on Connect web site if you want people to take seriously your words about you taking feedback seriously. I’d think dev division is an important enough part of Microsoft to be able to push changes in some web site, therefore the website doesn’t get improved because you guyes don’t care about that enough. Therefore…