Tag Archives: debug

Diving into the debugger again, we’re going to look at a very early feature that I had nothing to do with. Common to all programming languages (and at a deeper level, all computation) are data structures that contain other elements – “collections”. These may be arrays, lists, sets, mappings, or any other number of simple or complex types, each with their own preferred access method and characteristics.

What do I mean by “preferred access method”? In short, the most efficient manner to retrieve the element that you want. For example, arrays are best accessed using the element index, while (linked) lists prefer accessing elements adjacent to another. Sets are best used for mapping a value to true or false (as in, “does the set contain this element?”), and so on.

However, when it comes to debugging and introspection on a collection, the most useful access is complete enumeration. Retrieving every value in the collection and displaying it to the user allows many issues to be debugged that would otherwise be difficult or impossible. Those who are used to debugging in Visual Studio or another IDE will be familiar with the ability to view the contents of lists, sets and even list-like objects with no memory store (an Enumerable, in .NET):

Like .NET, Python has a range of collection types, including some that can be infinitely large without actually being stored in memory. Being able to view the elements in these collections is essential when debugging, so much so that Python Tools for Visual Studio had support in its earliest public releases.

This post will look at the messages sent between the debugger and debuggee, how the collection contents are exposed to the user, and the approaches used to manage unusual collection types. The code is from changeset baff92317760, which is an early implementation and some changes have been made since.

Debugger Messages

As we saw in my earlier blog on User-unhandled Exceptions, there are a number of messages that may be sent between the debugger (which is a C# component running within Visual Studio) and the debuggee (a script running in a Python interpreter). The command sent is CHLD and it is handled in visualstudio_py_debugger.py like this:

The two important parts to this handler are text and the enum_child_on_thread function. text is sent from the debugger and specifies the expression to obtain children for. This expression is compiled and evaluated in the context of the active call stack, which lets the user specify whatever they like in the Watch or Immediate windows, provided it is valid Python code.

We will look at enum_child_on_thread later, but it will eventually call report_children in order to send the results back to the debugger:

The _pendingChildEnums is part of the stateless communication infrastructure that allows the UI to remain responsive while the results are being collected. An array res is created to contain the result and pass it to the visualiser.

Displaying Children

Visual Studio provides a number of tool windows that display variable values. The one shown in the image at the top of this post is the Locals window, which displays variables in the active scope. There is also the Watch window, which lets the user enter any expression they live, the Parallel Watch window, which does the same thing across multiple threads, and the Autos window, which chooses relevant variables automatically.

For all of these views, the values that are displayed are implementations of the IDebugProperty2 interface: in PTVS, the implementation is in AD7Property.cs. (These are passed out to VS from various places in the debugger which we won’t be looking at right now.) The method that is most relevant here is IDebugProperty2.EnumChildren:

This method is called when the variable is expanded in VS. The call to _evalResult.GetChildren performs the communication described earlier, and will block until the list of the collection contents is available or the timeout expires. New AD7Property instances are created for each returned expression, allowing them to also be displayed in the variable windows. If they are expandable, they can in turn be expanded and have their elements displayed.

The other method of interest is IDebugProperty2.GetPropertyInfo, which returns a filled DEBUG_PROPERTY_INFO structure consisting mostly of displayable strings. (In fact, AD7Property only implements one method other than these two, which is SetValueAsString. IDebugProperty2 really does only serve a single purpose.) These strings are what are displayed in Visual Studio:

Looking into Collections

Now that we’ve seen how the debugger and debuggee communicate with each other, and how the debugger communicates with Visual Studio, let’s have a look at obtaining the values of the collections.

The actual work is performed in visualstudio_py_debugger.py in the enum_child_locally function. Obtaining the members is surprisingly simple, since the debugger is wrritten in Python and all supported collections have a consistent iteration interface:

Dictionaries require special handling, since normal enumeration only includes keys and not values, but all other iterable objects can be passed directly to enumerate. Both cases of the snippet shown result in enum containing a sequence of key-value tuples, where the key for non-dictionary collections is the 0-based index. The section of code following this converts the sequence to a list.

A large amount of exception handling exists to safely handle ‘bad’ objects. Any developer can implement their own __iter__(), __next__() (neenext()) or items() methods, potentially causing issues in our code. When an exception is thrown, we either skip that element (which also skips that index) or simply abandon enumeration completely. However, there are two types of ‘collections’ that need special handling.

The first are infinite iterators, which are a “list-like object with no memory store” like we saw earlier. When one of these is encountered, there is no way to discover it in advance. A timeout is used to prevent Visual Studio from waiting forever. However, because of the stateless nature of the debugger/debuggee communication, even though VS has decided it is no longer interested, the Python code will continue creating an infinitely long list of objects to return (until a MemoryError is raised, and then the list and the strings are deallocated and it’s as if nothing ever happened).

A relatively simple fix is used here: the number of elements returned is capped at 10,000. Because it is completely unwieldly to view 10,000 elements in the variable windows, most users will never encounter this limit. At the same time, fewer users are unlikely to see iterables displaying no elements, and infinite iteration errors are often identifiable from a short subsequence that would otherwise not be displayed. So while it looks like a hack, the end result is a better experience. (And you can view any element you like by adding ‘[<index>]’ after the name in the Watch window, even well beyond the 10,000 element limit.)

The other ‘collection’ that receives special handling are objects themselves. You have probably noticed throughout the code samples and SDK documentation that the “expandable” objects contain “children”. This functionality is not just used for collections, but also to allow objects to expand and list their members. If all the code above fails with an exception (which will typically be TypeError when attempting to iterate a non-iterable object), the following code runs:

This uses Python’s dir function to obtain a list of all the members of the object, filters out private (__special_name__) members and callable methods, and returns the rest in a similar fashion to the members of a dictionary.

Summary

Debugger support for enumerating collections allows Python Tools for Visual Studio to display members of collections in the user’s code. Visual Studio provides the user interface for the functionality, requiring only minimal implementation on the part of the language-specific debugger. As well as collections, PTVS (and other VS languages) use this to expand regular objects and display their member variables. This feature was part of the very first releases of PTVS.

This feature is one that I added to Python Tools for Visual Studio as an intern in 2011. Those familiar with debugging in Visual Studio will know that if your code throws an exception that is not handled, the debugger breaks at the statement that caused the error:

This feature is known as a user-uncaught exception and is incredibly useful in debugging, since it provides an opportunity to inspect local state before the stack unwinds. It is optional but defaults to on for most exceptions:

The “thrown” column breaks before the system traverses the stack looking for a handler (the “first chance”), while the “user-unhandled” column breaks after traversal if no code chooses to handle it, but before the stack actually begins to unwind (the “second chance”).

In Python, an unhandled exception will unwind the stack immediately, executing finally blocks and otherwise destroying state as it goes, printing a basic trace at the end. In other words, the first traversal step does not exist in Python and there is no second chance. Before adding this feature, PTVS only supported breaking on the first chance:

The aim of this feature was to mimic other languages by simulating the first traversal of exception handlers without modifying the program state. Step one of this task was to enable the UI, step two was to find a way to detect active exception handlers, and step three was to extend the debugger to support the feature.

To follow along at home, the changeset is baff92317760, which I will quote from (and link to) where relevant. (The code has changed again since this commit, but these are the relevant ones for this discussion.) I also recorded a short video around the time this feature was added to demonstrate how it works.

Enabling the UI

Enabling the check boxes for user-unhandled exceptions is simply a case of changing the registration. Each debugging engine can list the exceptions it supports under AD7Metrics\Exception\{engine_guid}\{exception_name}. This allows the user to select whether or not to break on these exceptions, while the debugging engine is still fully responsible for handling the break itself.

Within these keys is a State value that combines values from theEXCEPTION_STATE enumeration. The values we used are EXCEPTION_STOP_USER_UNCAUGHT and EXCEPTION_JUST_MY_CODE_SUPPORTED (0x4020):

In PTVS, the exceptions are registered using the ProvideDebugException attribute on PythonToolsPackage.cs, which includes a state parameter for setting this value. By modifying ProvideDebugExceptionAttribute.cs we can simply change the default for all of our exceptions:

After adding this line to the attribute constructor and rebuilding, the dialog has the checkboxes enabled:

However, simply making the option available does not provide any functionality, since it is entirely managed by Visual Studio. The options selected by the user are exposed to the debugger through the IDebugEngine2.SetException method, implemented in AD7Engine.cs. The following changes were made to support both forms of exception handling:

One trick here is that this method is only called where a setting is different from its default, which means that the debug engine has to be aware of all the default values. For example, we made AttributeError not break by default, since it is often handled in a way we cannot detect, but now SetExceptionInfo only receives a notification for when handling is enabled, and so the debug engine has to be aware that unless it is told otherwise, it should not break on AttributeError. The changes required to make it break when the exception is thrown are discussed later.

Detecting Exception Handlers

As described above (briefly), Python uses a different approach to thrown exceptions than other languages. The main difference is that exception handlers are checked as the stack unwinds, so by the time Python code can be sure that an exception is not handled, the stack is gone, all finally blocks have been run and only the traceback remains. The traceback contains source files and line numbers, but no information about local variables or values. Ideally, the debugger would detect that an exception in unhandled without unwinding and break in before the call stack is destroyed.

Since we are restricted to static analysis, a fallback position is necessary. The implementation assumes that all exceptions are unhandled unless proven otherwise, which makes us more likely to break when an exception occurs. The alternative would result in more stack traces and less breaking. Since the developer is already debugging their code, it is fair to assume that a false positive (breaking when a handler exists) is preferable to a false negative (not breaking when there is no handler).

In terms of implementation, the GetHandledExceptionRanges method in PythonProcess.cs does the main detection work:

Given a filename, we parse the file and produce a syntax tree. With TryHandlerWalker we simply collect all the try nodes, each of which also contains the list of caught expressions and, importantly, the line numbers of the try statement and the first except statement. This information is sent to the debugger in response to a REQH command (discussed below).

Later, the debugger will parse expressions from the except statements to compare against the thrown exception. Line numbers are sufficient to determine whether code is in a try block (since try and except are only valid at the start of a line) but anything can be specified as the except expression. For example, the following is working (perhaps not valid, and certainly not easy to read) code:

try:
do_something()except get_exception_types()[1]:print("handled")

try:
do_something()
except get_exception_types()[1]:
print("handled")

When we are simulating a stack unwind, we cannot call the function to find out what it returns, firstly because we are not in the correct call frame and secondly because of potential side effects. (The closest thing to a sensible use of a call as an except expression is logging, which would mean that if we called it we would produce spurious messages.)

To handle as many safe cases as possible, the debugger will only look up names and modules. As soon as anything more complicated appears in an expression list, we assume that it does not handle the current exception and keep looking for another handler. Because we have access to the call frames, we are able to look up names in the correct scope. For example, in the following code we will determine that the exception has a handler, even though its name (zde) is defined in a different call frame than where the exception is thrown:

An issubclass call is used to determine whether the exception will be caught by each possible handler.1 When it returns true (or a plain except: is encountered), execution is continued and the debugger is not notified of the exception. If we reach a call frame that does not have code available (meaning we can’t get the handlers) or that belongs to the debugger itself, we assume the exception is unhandled and break.

1Of course, this means that arbitrary code could still be executed in the form of a __subclasscheck__ method. This method should not have side effects anyway, so all we’d really achieve by calling it is to expose a bug earlier.

Extending the Debugger

In order for the debugger to support the updated exception handling mechanism, we have to add the new REQH (REQuest Handlers) and SEHI (Set Exception Handler Info) commands. Commands are used for stateless communicate between the Python process being debugged and the Visual Studio instance doing the debugging. REQH will be sent from the debuggee to request the list of exception handlers in a particular source file and SEHI is sent with the response. PythonProcess.cs contains the handling for REQH events:

The GetHandledExceptionRanges method was shown above; HandleRequestHandlers calls this method and transmits the line numbers and exception types to the debuggee. The implementation on the debuggee side is in visualstudio_py_debugger.py. The handling for the SEHI command looks like this:

Notice that the filename is sent both ways. Because the protocol is stateless, the debuggee cannot automatically associate the response with its request. In practice, it will block until it receives the response it expects, but it would require no protocol modifications to support preemptively sending SEHI commands. To avoid parsing source files repeatedly, the debuggee caches the handler lists, which are stored as tuples containing the same information found by GetHandledExceptionRanges.

Breaking on an exception is relatively straightforward. The handler for when an exception is thrown already had a ShouldBreak test, which was changed to check the mode for the exception as, if necessary, whether any handlers exist.

The IsHandled method performs the call-stack traversal described earlier, returning False at the first frame that has no code available or True at the first frame that has a handler matching the exception type:

The search starts from the first frame in the traceback, which is the frame where the exception was caused, and goes backwards through f_back (that is, towards the caller) searching each file’s handler list. If no handlers are cached, they are requested from the debugger, and the linear search and use of line ranges ensures the correct behaviour for nested try blocks.

Summary

This feature added the ability for Python Tools for Visual Studio to detect and break on unhandled exceptions without unwinding the stack. This allows developers to inspect the state of their program and not just the stack trace. Handler detection is based on simple source code analysis, performed when an exception has been thrown, and assumes that if it can’t guarantee that the exception has a handler then it should break. It was first released in PTVS 1.0, August 2011.