What is key about this PoC is that is represent the real-time loop (almost a REPL) that we need for (some types) of security issues.

Next we need to think about the best way to present this information to developers. For example I was thinking that we might want to have a number of colours shown depending on the type of issue, how serious it could be, its exploitability, etc...

Another cool idea is to change the cursor's color or its shape (think bigger or smaller), depending on the number of issues currently outstanding :)

How about a big red strikethrough on code that is insecure. Mouseover: "Don't even think about compiling this" ;)
–
SteveJun 21 '12 at 23:24

1

I would want it to have its own window. I would want this to be perhaps listed in a similar fashion as the "TODO" and/or Bookmarks. I would want an options to highlight and change the visual appearence of the code as purely an option.
–
RamhoundJun 22 '12 at 12:39

3 Answers
3

How about triggering a security compile when the window loses focus or gains focus? Or after 10 seconds of user inaction? Definitely do it when the file is saved, that's a good one that always happens pre-build.

For developer feedback, how about displaying colored skulls and crossbones in the left side of the pane, in the area next to where the debugging breakpoints get displayed. Color them to indicate severity: something like yellow for "must fix", and red for "OMG!!!" Hovering the mouse pointer over them could pop up a tool tip describing the bug.

Maybe set the background to a light red (or colored) gradient, with the intensity centered on the vulnerability. Or use colored bars alongside the scroll bar, like WinDiff's screen for showing you where differences are. Use black, white, or non-existent line segments for no bug regions, and line segments that color match the skulls representing the vulnerabilities.

At the very least make sure it's outputting errors that break the build.

You need to tell the developer that no automated system can find every flaw, and the the developer needs to be on their toes and look for flaws on their own. A false sense of security is the absolute |worst| thing you could provide to a developer.

Trust nothing, test everything.

(On a side note, simply highlighting every dangerous function and providing documentation of why it is dangerous could be very useful. Also, you should recognize you will never be able to solve 1/2 of the serious security problems in a real application with this "solution". The lack of an important control cannot be underlined or highlighted, because doesn't exist.)

This is a very cool idea, but I think that to succeed it is important to integrate this seamlessly with the existing environment.

To that end, I would suggest NOT to come up with some new, unexpected way to present the information, but to use the existing formats and mechanisms.

For example, underline critical and high-risk vulnerabilities with a red squiggly line - just as Visual Studio does for compilation errors by default, and just as almost all programmers expect. You could mark less serious flaws with less scary colors - blue, green, etc.
I would go one step further - and allow this to be configurable, int VS's Fonts and Colors options dialog, to enable programmers to either have it fit in with their specific settings, or even have it stand out. You could have one setting for Critical, another color for High, etc.

Additionally, just as with compilation errors and warnings, there should be a window listing all currently found issues - the richer this window, the better, of course (filtering, sorting, etc).

This would go a great distance in training developers that security flaws, are bugs just like functional bugs - and, in some cases (e.g. Critical/High), should be treated as compilation errors.

Note that VS's builtin Code Analysis works similar to this (except not realtime, for the most part).