Saturday, June 16, 2007

Why can't the pydev debugger work with turbogears?

Ok, there's a problem that can't really be overcome in the pydev debugger when using turbogears... (just doing "import turbogears" would already break it).

Actually, no OPTIMIZED debugger would be able to work with that. I'm saying optimized because the implementation seems to take into account naive debuggers which would trace all the calls within all the frames (pydev only traces frames with breakpoints).

The problem is: there's a module that turbogears uses (in my tests: DecoratorTools-1.4-py2.5.egg) which has a decorator named: decorate_assignment. This decorator uses the tracing facility that python provides for debuggers and removes the current debugger tracer function. It still tries to restore it if it was tracing the frame previously (but that would hardly ever happen in an optimized debugger).

So, there's no way to actually fix that from pydev, but there are some options to make it work:

1. Using the pydev extensions remote debugger (but if that decorator is called after the remote debugger is set, the debugger would stop working again, so, this option would only useful if that decorator is not used later).

2. Removing that decorator from the places that use it in turbogears (the implications for that would have to be checked).

3. Hard-coding it to return the pydev tracing function. To do that, the file: DecoratorTools-1.4-py2.5.egg\peak\util\decorators.py must be changed so that the function "def decorate_assignment(callback, depth=2, frame=None):" does not use the call:

The 3rd option is probably the easier in the short run for those wanting to debug turbogears in pydev, but I think that the 2nd should be the one actually used (as a general rule, I believe that only debuggers should play with the tracing facility, because it tends to bee way to instrusive, and it's probably the most un-optimized way of doing something, as you're going to trace all that happens, which can lead to a large overhead).

Hello I tested successfully the 3rd option with python-2.4, DecoratorTools-1.4 and pydev 1.3.5But I have upgraded to python-2.5, DecoratorTools-1.5 and pydev 1.3.8 and now, my application is not working when running in the debugger.I get the pydev warning about usage of sys.settrace() and the application start, I can click some link, but I looks like when I'am submiting forms, I get error:TypeError: ("'NoneType' object is not callable", <bound method Root.quick_login of <myapps.controllers.Root object at 0xa92ebac>>)Any idea ?

URL: http://eg01.apps.loc:8080/eg/quick_login?user_name2=managerFile '/kolab/lib/python/site-packages/Paste-1.4-py2.5.egg/paste/evalexception/middleware.py', line 308 in respond return_iter = list(app_iter)File '/kolab/lib/python/site-packages/CherryPy-2.2.1-py2.5.egg/cherrypy/_cpwsgi.py', line 75 in wsgiApp environ['wsgi.input'])File '/kolab/lib/python/site-packages/CherryPy-2.2.1-py2.5.egg/cherrypy/_cphttptools.py', line 72 in run self._run()File '/kolab/lib/python/site-packages/CherryPy-2.2.1-py2.5.egg/cherrypy/_cphttptools.py', line 105 in _run self.main()File '/kolab/lib/python/site-packages/CherryPy-2.2.1-py2.5.egg/cherrypy/_cphttptools.py', line 254 in main body = page_handler(*virtual_path, **self.params)File '<string>', line 3 in quick_loginFile '/kolab/lib/python/site-packages/TurboGears-1.0.3.2-py2.5.egg/turbogears/controllers.py', line 340 in expose _build_rules(func)File '/kolab/lib/python/site-packages/TurboGears-1.0.3.2-py2.5.egg/turbogears/controllers.py', line 246 in _build_rules for ruleinfo in func._ruleinfo:File '/kolab/lib/python/site-packages/TurboGears-1.0.3.2-py2.5.egg/turbogears/controllers.py', line 246 in _build_rules for ruleinfo in func._ruleinfo:TypeError: ("'NoneType' object is not callable", <bound method Root.quick_login of <apps.controllers.Root object at 0x9ab3ecc>>)

The thing breaks correctly, and I can step, but the Eclipse debugger doesn't actually catch exceptions in the code. Apache's stderr says:

%stderr

PYDEV DEBUGGER WARNING:sys.settrace() should not be used when the debugger is being used.This may cause the debugger to stop working correctly.If this is needed, please check: http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.htmlto see how to restore the debug tracing back correctly.Call Location: File "/usr/local/eclipse/plugins/org.python.pydev.debug_1.3.10/pysrc/pydevd.py", line 743, in settrace sys.settrace(debugger.trace_dispatch)

Actually, calling sys.settrace still works (it won't actually stop the execution when it happens... it just prints it to the screen).

You can use pydevd_tracing.SetTrace to call the original tracing without having a warning message.

But aside from that, you must still restore the tracing facility as specified in the post for the debugger to keep working (and it won't be able to debug code that happens when the tracing function is replaced).

I notice you have code that's checking sys.settrace and issuing a warning; but if settrace() is being passed one of *your* trace functions (i.e., it's restoring the tracer, not replacing it), then why not just call the old settrace() with the correct function?

What I mean is, you could safely restore PyDev's global trace function if settrace() is called with None, or with a local trace function. In other words, tracing a DecoratorTools-using app (or anything else that uses a similar technique) would then work correctly, except for the warning(s).

Just in case this is still an issue, Python 2.6 provides a sys.gettrace() function which means that instead of unconditionally trying to reset the settrace function to the pydev debugger tracer all you need to do in decorator tools is change the line:-

oldtrace = [frame.f_trace]

to

oldtrace = [sys.gettrace()]

and debugging should work again. This works fine with the latest version of decorator tools and turbogears 1.1. The solution originally outlined in this post won't work with versions of turbogears greater than 1.0.4 because of the way decorator assignment is being used in the peak Rules package.

Thanks for the tip... as it happens, I was able to implement a refactoring that might actually fix this for older Pythons as well (but definitely in 2.6). It's in SVN if you'd like to give it a try; I'd like some feedback before pushing it out as an official release.

I've found this issue, but turbogears is not installed in my virtualenv.

PYDEV DEBUGGER WARNING:sys.settrace() should not be used when the debugger is being used.This may cause the debugger to stop working correctly.If this is needed, please check: http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.htmlto see how to restore the debug tracing back correctly.Call Location: File "/usr/lib/python2.7/bdb.py", line 225, in set_trace sys.settrace(self.trace_dispatch)