Re: [Webware-devel] [UPDATE] Autoreload feature in CVS

On Fri, 2002-10-25 at 01:40, Ian Bicking wrote:
> Looks good. I changed it a bit so it would also monitor modules that
> were loaded before ImportSpy was started up (e.g., Application.py). It
> would be good to also add any configuration files -- I suppose a hook
> could be added to Configurable.
Good idea!
> To do that ImportSpy should probably
> not auto-install itself, since Configurable would have to check if
> ImportSpy was on.
Well right now ImportSpy hooks itself in regardless of whether
AutoReload is activated or not. I would have preferred to have
AutoReloadingAppServer install the ImportSpy hook, but I also wanted
ImportSpy to get hooked in as soon as possible, so that most modules get
monitored. Do you see a better way of doing this?
> Another feature that would be nice would be adding checking Python
> standard libraries. On Unix that will generally be files starting in
> /usr/lib/pythonX.X -- but that's obviously not general, even on Unix.
> I'm not sure how to identify that directory. It might be
> os.path.join(sys.prefix, 'lib/python%i.%i' % (sys.version_info[0],
> sys.version_info[1])).
Could we use access() to check for write permissions? Standard python
modules would be installed without write permissions for most users.
But then again, people might run their appservers under a user which
doesn't have write permissions on any of their servlets, for security
reasons. I don't see a good general way of solving this.
> I did have a problem with the actual restart, though. It shuts down and
> I get:
> /usr/bin/python: can't open file 'Launch.py'
>
> I don't get it -- I'm starting it up from ./AppServer (in the WebKit
> directory). $PWD is the WebKit directory. Launch.py is in the WebKit
> directory. So I don't know why it isn't working.
Hmm... Launch.py changes the working directory to the Webware directory
(one level up from WebKit), so that's probably why autoreload doesn't
work in this case. If you called AppServer with the full path it would
likely work. Is there a good way to fix this? I guess we could
translate the value of sys.argv[0] to take the chdir into account when
the appserver is called with a relative pathname.
> On a stylistic note: it's best to use """ instead of ''' for docstrings
> -- Emacs sometimes is confused with '''. Also please be sure to put
> whitespace after all commas.
Thanks for the tips. I remembered seeing something about """ vs ''',
but couldn't rememebr which was preferred. A grep of the code in CVS
wasn't conclusive. :)
--
Jason D. Hildebrand
jason@...

Thread view

Earlier this afternoon I committed an auto-reload feature to Webware
CVS. I'd like to credit Tavis Rudd, who provided the initial
implementation for this feature. Here are the details:
Overview
--------
The purpose of this feature is to make life easier on the developer by
having the appserver notice when you make modifications to source files
and restart itself to pick up these changes.
Details
-------
Source files which are monitored include all *.py files loaded by the
application server, which includes servlets, __init__.py files in
package directories, and the Webware sources themselves.
The monitor knows about PSPs and Cheetah templates, and will monitor the
source file corresponding to the generated servlet or template file,
which is what you want. Note: I've only tested PSPs; please let me know
if Cheetah templates also work as expected.
The server will also monitor files which could not be successfully
imported. For example, say you write MyPage.py which contains a syntax
error, and then try to request the page. Because of the syntax error
the module cannot be imported, which will give you a traceback. But
after error and save the file, the appserver will restart itself to pick
up the change.
If you install python-fam (http://python-fam.sf.net) and have a
libfam/famd (file alteration monitor) installed on your machine, this
will be used, which is much more efficient. Otherwise the appserver will
poll the files at regular intervals for changes. The method of
monitoring (FAM or polling) is printed by the server at startup.
The server is restarted using exec, which causes the existing process to
be replaced. This means that it will work regardless of how you have
your permissions set up and under what user the appserver executes in
your environment. It will be called with the same command line
arguments as the original invocation.* The process keeps the same
process id (PID), too.
* Well, actually occurrences of "restart" are replaced by "start".
Since the server shuts itself down before starting itself again, this is
what you want.
Configuration
-------------
Autoreload is disabled by default. To enable it include the line
'AutoReload': 1,
in your AppServer.config file.
If you wish to customize the polling interval, you can also set:
'AutoReloadPollInterval': 5,
to check files every 5 seconds. The default is to check every 1 second.
Please let me know of any problems.
--
Jason D. Hildebrand
jason@...

Looks good. I changed it a bit so it would also monitor modules that
were loaded before ImportSpy was started up (e.g., Application.py). It
would be good to also add any configuration files -- I suppose a hook
could be added to Configurable. To do that ImportSpy should probably
not auto-install itself, since Configurable would have to check if
ImportSpy was on.
Another feature that would be nice would be avoiding checking Python
standard libraries. On Unix that will generally be files starting in
/usr/lib/pythonX.X -- but that's obviously not general, even on Unix.
I'm not sure how to identify that directory. It might be
os.path.join(sys.prefix, 'lib/python%i.%i' % (sys.version_info[0],
sys.version_info[1])).
I did have a problem with the actual restart, though. It shuts down and
I get:
/usr/bin/python: can't open file 'Launch.py'
I don't get it -- I'm starting it up from ./AppServer (in the WebKit
directory). $PWD is the WebKit directory. Launch.py is in the WebKit
directory. So I don't know why it isn't working.
On a stylistic note: it's best to use """ instead of ''' for docstrings
-- Emacs sometimes is confused with '''. Also please be sure to put
whitespace after all commas.
Ian

On Fri, 2002-10-25 at 01:40, Ian Bicking wrote:
> Looks good. I changed it a bit so it would also monitor modules that
> were loaded before ImportSpy was started up (e.g., Application.py). It
> would be good to also add any configuration files -- I suppose a hook
> could be added to Configurable.
Good idea!
> To do that ImportSpy should probably
> not auto-install itself, since Configurable would have to check if
> ImportSpy was on.
Well right now ImportSpy hooks itself in regardless of whether
AutoReload is activated or not. I would have preferred to have
AutoReloadingAppServer install the ImportSpy hook, but I also wanted
ImportSpy to get hooked in as soon as possible, so that most modules get
monitored. Do you see a better way of doing this?
> Another feature that would be nice would be adding checking Python
> standard libraries. On Unix that will generally be files starting in
> /usr/lib/pythonX.X -- but that's obviously not general, even on Unix.
> I'm not sure how to identify that directory. It might be
> os.path.join(sys.prefix, 'lib/python%i.%i' % (sys.version_info[0],
> sys.version_info[1])).
Could we use access() to check for write permissions? Standard python
modules would be installed without write permissions for most users.
But then again, people might run their appservers under a user which
doesn't have write permissions on any of their servlets, for security
reasons. I don't see a good general way of solving this.
> I did have a problem with the actual restart, though. It shuts down and
> I get:
> /usr/bin/python: can't open file 'Launch.py'
>
> I don't get it -- I'm starting it up from ./AppServer (in the WebKit
> directory). $PWD is the WebKit directory. Launch.py is in the WebKit
> directory. So I don't know why it isn't working.
Hmm... Launch.py changes the working directory to the Webware directory
(one level up from WebKit), so that's probably why autoreload doesn't
work in this case. If you called AppServer with the full path it would
likely work. Is there a good way to fix this? I guess we could
translate the value of sys.argv[0] to take the chdir into account when
the appserver is called with a relative pathname.
> On a stylistic note: it's best to use """ instead of ''' for docstrings
> -- Emacs sometimes is confused with '''. Also please be sure to put
> whitespace after all commas.
Thanks for the tips. I remembered seeing something about """ vs ''',
but couldn't rememebr which was preferred. A grep of the code in CVS
wasn't conclusive. :)
--
Jason D. Hildebrand
jason@...

On Fri, 2002-10-25 at 10:48, Jason Hildebrand wrote:
> On Fri, 2002-10-25 at 01:40, Ian Bicking wrote:
> > Looks good. I changed it a bit so it would also monitor modules that
> > were loaded before ImportSpy was started up (e.g., Application.py). It
> > would be good to also add any configuration files -- I suppose a hook
> > could be added to Configurable.
>
> Good idea!
>
> > To do that ImportSpy should probably
> > not auto-install itself, since Configurable would have to check if
> > ImportSpy was on.
>
> Well right now ImportSpy hooks itself in regardless of whether
> AutoReload is activated or not. I would have preferred to have
> AutoReloadingAppServer install the ImportSpy hook, but I also wanted
> ImportSpy to get hooked in as soon as possible, so that most modules get
> monitored. Do you see a better way of doing this?
Well, all modules that are loaded get added now, even if ImportSpy
hadn't been started yet. So it's not a problem if it's slow to start
up.
> > Another feature that would be nice would be adding checking Python
> > standard libraries. On Unix that will generally be files starting in
> > /usr/lib/pythonX.X -- but that's obviously not general, even on Unix.
> > I'm not sure how to identify that directory. It might be
> > os.path.join(sys.prefix, 'lib/python%i.%i' % (sys.version_info[0],
> > sys.version_info[1])).
>
> Could we use access() to check for write permissions? Standard python
> modules would be installed without write permissions for most users.
> But then again, people might run their appservers under a user which
> doesn't have write permissions on any of their servlets, for security
> reasons. I don't see a good general way of solving this.
Yeah, I guest that won't work. That last idea I gave (os.path.join...)
might be correct, though -- it matches all the directories Stefan gave
from distutils.
> > I did have a problem with the actual restart, though. It shuts down and
> > I get:
> > /usr/bin/python: can't open file 'Launch.py'
> >
> > I don't get it -- I'm starting it up from ./AppServer (in the WebKit
> > directory). $PWD is the WebKit directory. Launch.py is in the WebKit
> > directory. So I don't know why it isn't working.
>
> Hmm... Launch.py changes the working directory to the Webware directory
> (one level up from WebKit), so that's probably why autoreload doesn't
> work in this case. If you called AppServer with the full path it would
> likely work. Is there a good way to fix this? I guess we could
> translate the value of sys.argv[0] to take the chdir into account when
> the appserver is called with a relative pathname.
Yes, that sounds right. But the environmental variable PWD remains the
same... but I guess that doesn't matter. I guess we'll just have to
track the original directory, and do a chdir before restarting.
Ian

Hi Ian
On 25 Oct 2002, Ian Bicking wrote:
> Another feature that would be nice would be avoiding checking Python
> standard libraries. On Unix that will generally be files starting in
> /usr/lib/pythonX.X -- but that's obviously not general, even on Unix.
> I'm not sure how to identify that directory. It might be
> os.path.join(sys.prefix, 'lib/python%i.%i' % (sys.version_info[0],
> sys.version_info[1])).
Perhaps it's best to use the distutils package:
>>> import pprint
>>> from distutils import sysconfig
>>> pprint.pprint(sysconfig.get_config_vars())
This prints a lot of settings which might be interesting. Among them are
'BINLIBDEST': '/usr/local/lib/python2.2',
'DESTDIRS': '/usr/local /usr/local/lib /usr/local/lib/python2.2 /usr/local/lib/python2.2/lib-dynload',
'DESTLIB': '/usr/local/lib/python2.2',
'DESTSHARED': '/usr/local/lib/python2.2/lib-dynload',
'LIBDEST': '/usr/local/lib/python2.2',
'LIBP': '/usr/local/lib/python2.2',
However, I don't know the difference between DESTLIB and LIBDEST ;-/ ,
maybe they are documented somewhere.
You can get individual variables with
>>> sysconfig.get_config_var('LIBDEST')
'/usr/local/lib/python2.2'
Stefan

On Friday 25 October 2002 08:48 am, Jason Hildebrand wrote:
> Well right now ImportSpy hooks itself in regardless of whether
> AutoReload is activated or not. I would have preferred to have
> AutoReloadingAppServer install the ImportSpy hook, but I also wanted
> ImportSpy to get hooked in as soon as possible, so that most modules
> get monitored. Do you see a better way of doing this?
Just a thought: Could the top of Launch.py import the sys module and
check the command line args to determine if ImportSpy should be loaded?
This would obviously require a command line arg either to AppServer and
perhaps a convenience script named AutoReloadingAppServer script.
I'm not sure if ImportSpy has any performance impact on
non-autoreloading servers (e.g., your non-autoreloading production box).
-Chuck

On Sun, 2002-11-10 at 02:27, Chuck Esterbrook wrote:
> On Friday 25 October 2002 08:48 am, Jason Hildebrand wrote:
> > Well right now ImportSpy hooks itself in regardless of whether
> > AutoReload is activated or not. I would have preferred to have
> > AutoReloadingAppServer install the ImportSpy hook, but I also wanted
> > ImportSpy to get hooked in as soon as possible, so that most modules
> > get monitored. Do you see a better way of doing this?
>
> Just a thought: Could the top of Launch.py import the sys module and
> check the command line args to determine if ImportSpy should be loaded?
>
> This would obviously require a command line arg either to AppServer and
> perhaps a convenience script named AutoReloadingAppServer script.
>
> I'm not sure if ImportSpy has any performance impact on
> non-autoreloading servers (e.g., your non-autoreloading production box).
This has been changed since then -- now the hook is only put in when
auto-reloading is on (so there's no overhead).
Ian

On Sunday 10 November 2002 11:26 am, Ian Bicking wrote:
> On Sun, 2002-11-10 at 02:27, Chuck Esterbrook wrote:
> > On Friday 25 October 2002 08:48 am, Jason Hildebrand wrote:
> > > Well right now ImportSpy hooks itself in regardless of whether
> > > AutoReload is activated or not. I would have preferred to have
> > > AutoReloadingAppServer install the ImportSpy hook, but I also
> > > wanted ImportSpy to get hooked in as soon as possible, so that
> > > most modules get monitored. Do you see a better way of doing
> > > this?
> >
> > Just a thought: Could the top of Launch.py import the sys module
> > and check the command line args to determine if ImportSpy should be
> > loaded?
> >
> > This would obviously require a command line arg either to AppServer
> > and perhaps a convenience script named AutoReloadingAppServer
> > script.
> >
> > I'm not sure if ImportSpy has any performance impact on
> > non-autoreloading servers (e.g., your non-autoreloading production
> > box).
>
> This has been changed since then -- now the hook is only put in when
> auto-reloading is on (so there's no overhead).
Ah. I saw "import ImportSpy" at the top of Launch.py which made me
think it was required and doing something. I then went reading and
found the above message.
Launch.py seems to work fine without it, even when AutoReload is on, so
I have removed it.
-Chuck