Introduction

The following was posted to the comp.lang.python newsgroup on 24-May-2001. Since it explains very well the options for keeping the GUI responsive when the application has to do long running tasks I decided to copy it here verbatim. Many thanks to David Bolen for taking the time and effort to compose and post this message.

--Robin

The Question

Daniel Frame writes:

Basically, I've constructed a GUI with wxPython which runs a very long function when a 'START' button is pressed. Of course, the GUI is 'locked up' until this function completes itself. I was wanting to add a 'STOP' button to my application which would halt this other function if the user got tired of waiting for it to complete. Is it possible to accomplish this without resorting to using threads? If so, How can I have my running function check occasionally to see if the stop button was pressed? I've heard of wxYield, but I can't seem to implement it properly.

David's Response

There's really three possibilities I would think of - threading, wxYield, or chunking up your processing in a wxEVT_IDLE handler. I've given a small sample of each below.

I personally like threads for this sort of thing (I think they leave the UI the most responsive), but there's no hard and fast rule. So whichever you're more comfortable with.

Threading really isn't that hard - you can just start up a thread to do your processing, and have it send an event to your main GUI thread when it has completed. While it is working, it can check an event object or some other flag variable to indicate that it should give up and stop.

For a simple sort of example, here's a small bit of code that presents a simple (really dumb visually) frame, and uses a worker thread to simulate some processing (that takes 10s, resulting in a value of 10), while permitting it to be aborted. This could be extrapolated to doing any sort of lengthy processing and returning any sort of result. Intermediate events could be generated during the processing to give some indication to the main GUI thread of how things were proceeding (perhaps updating something like a gauge or some other visual indicator).

Oh, and if you're concerned with hanging on an exit if your thread doesn't terminate for some reason, just add a "self.setDaemon(1)" to the init and Python won't wait for it to terminate.

The second approach, using wxYield, should be fine too - just add a call to wxYield() somewhere within the computation code such that it executes periodically. At that point, any pending window events will be dispatched (permitting the window to refresh, process button presses, etc...). Then, it's similar to the above in that you set a flag so that when the original code gets control after the wxYield() returns it knows to stop processing.

As with the threading case, since all events go through during the wxYield() you need to protect against trying to run the same operation twice.

Here's the equivalent of the above but placing the computation right inside the main window class. Note that one difference is that unlike with threading, the responsiveness of your GUI is now directly related to how frequently you call wxYield, so you may have delays refreshing your window dependent on that frequency. You should notice that this is a bit more sluggish with its frequency of a wxYield() each second.

And finally, you can do your work within an idle handler. In this case, you let wxPython generate an IDLE event whenever it has completed processing normal user events, and then you perform a "chunk" of your processing in each such case. This can be a little tricker depending on your algorithm since you have to be able to perform the work in discrete pieces. Inside your IDLE handler, you request that it be called again if you aren't done, but you want to make sure that each pass through the handler doesn't take too long. Effectively, each event is similar to the gap between wxYield() calls in the previous example, and your GUI responsiveness will be subject to that latency just as with the wxYield() case.

I'm also not sure you can remove an idle handler once established (or at least I think I had problems with that in the past), so the code below just establishes it once and the handler only does work if it's in the midst of a computation. [Actually, you can use the Disconnect method to remove an event handler binding, although there is no real need to do so as there is very little overhead if you use a guard condition as in the code below. --Robin]

Slight modification for recursive tasks

Jeff Grimmett adds:

The above three examples got me going in the right direction, but the actual worker task used is a little on the simple side for what I needed to do, which was to recurse through directories and use the same class for each recursion.

Do I need to tell you that it's a bad idea to recurse 100 threads at once? Highly entertaining to watch but hardly useful of the end goal is to collect data serially!

The trick I settled on was to seperate the thread from the task. The means of communicating the shutdown command was the first thing I tackled. I settled on the python threading module's Event class. This class is basically a flag, which i suppose would also work quite well as long as the scope is correct.

1importthreading 2 3# This is a global flag that we will manipulate as needed 4# to allow graceful exit from the recursive search. 5KeepRunning = threading.Event()

The next thing I needed to do was define the scope of the actual worker thread.

The thread initializes with a pointer to the output window - which in this case is a wxTextCtrl() object - and the starting directory. Like the above example, this thread is self-starting, but need not be.

The first big difference is in the run() method. In the above example all the work is actually done BY the run() method. In this case, we create a WorkerBee object and retain a handle on it. Exactly why will become clear in a moment.

The second difference is that instead of keeping our 'keep running' variable local, we now have it global in the form of the KeepGoing Event object. Again, the reason why becomes clear in the next part.

While not directly on topic, the Update() method is simply a convenience to deal with the output display. If you were going to stdout this would be entirely uncessesary.

====

The WorkerBee class is the main point of difference between this example and the previous. Instead of the thread object doing all the work, it starts up the WorkerBee object.

The WorkerBee class recursively scans two directories and compares the results using the filecmp Python module. The exact task is not important, but I had to use something

THIS is why we need to keep the thread object and the workerbee seperate. If the workerbee was built around the Thread class, we would have a SWARM of workerbees! However, in this case what we end up with is a controlled recursion into a directory tree, with frequent checks on the KeepRunning Event flag. The whole thing shuts down quite nicely and leaves the GUI VERY responsive.

More Tips

The LongRunningTasks wiki page doesn't mention using Queues. When people use this technique what is the set up? For example, is your "worker thread" pushing data into the queue and then a wx.Timer is polling it out with a non-blocking q.get_nowait()?

That's one way. Another is to call wxWakeUpIdle when the item is put in the Queue, and then have a EVT_IDLE handler that fetches items from the queue. That way the item is usually processed very soon after it is put there without having to have a higher frequency timer.

Another option is to not use the queue or wxPostEvent and just use wxCallAfter passing to it the callable and parameters to be called in the GUI thread. (It uses wxPostEvent internally.) The implementation and docs of wxCallAfter is in core.py.

For communication between secondary threads and the main thread, you may use wxEvtHandler::AddPendingEvent or its short version wxPostEvent. These functions have a thread-safe implementation so that they can be used as they are for sending events from one thread to another. However there is no built in method to send messages to the worker threads and you will need to use the available synchronization classes to implement the solution which suits your needs yourself.

So you do not need a Queue to send messages from the worker thread to the main/gui thread: just use AddPendingEvent in the worker and a normal event handler in the gui. However if you need to send messages from the main/gui thread to the worker thread the right way to do it is probably a Queue, with the gui thread calling put() and the worker regularly checking for messages with get_nowait(). You can *probably* get away with using a simple boolean flag like some examples on this page use without any extra locking thanks to the GIL (Global Interpreter Lock), but for anything more complicated than that you need locking (like the Event used in the previous example), and for more than one kind of "message" a queue is usually the simplest way to go. Even something as simple as a worker thread that can be paused, resumed and cancelled is IMHO simpler using a queue than using a bunch of Event and/or Condition objects (the worker can just .get() on the queue to pause).

See also Brian Kelley's demo (Process.py) which shows how to do a long task in a separate process (using wxProcess and wxExecute) rather than a thread. A separate process is like running a separate application - it can run full speed, whereas a thread might be slowed about 30% by Python's global interpreter lock (I don't know which version you were using, but at least for the first few threads, Python does pretty well, resulting in fairly minimal overhead - Josiah).

MVoncken suggests: I prefer using a generator,(variant on wxYield,Example2):

While using a generator and wx.Yield() works, you are limited to having a single long-running-task at any time. This may be a serious limitation to your application. Instead, you can use the built-in wx.FutureCall() function to schedule as many calls as you want/need. I use this in to handle 'replace all' in one of my applications.

Alternatively, since wx.FutureCall creates and manipulates wx.Timer instances, we could just create a registry of stuff to call when a timer expires, and allow insertion of items to call into this registry. Why do we use a Timer/FutureCall instead of wx.CallAfter? It's mostly a matter of taste, but we use self.IsRunning() rather than keeping explicit state as a convenience.

Redirecting text from stdout to a wx.TextCtrl

I thought I might add this to the discussion. There are probably more ways than one to do this but... This is some sample code that I put together that updates a text control in wxpython from output being sent to stdout through backend "print" statments being executed in a separate thread.

The idea here is to that you're creating a class with a "write" method to rebind to stdout (the real value of stdout is always kept in sys.stdout for rebinding). Everytime a message is spit out to stdout an event is thrown out to the text control for processing. The problem is that the text control doesn't update itself. Therefore we need to get the control to process its pending events. Therefore we need to set up a timer that gets the text control to attempt to process its pending events every so often.

The threading code is pulled from Python in a Nutshell pretty much (which is what I'm using). It uses 2 Queues to process events. requestID, callable, args, kwds = self.requestQ.get() will suspend the thread untill the callable returns, and when it completes self.resultQ.put((requestID, callable(*args, **kwds))) will tell us that the process has finished its task. When this is done we submit an event to the GUI telling us that we are done.

There's better ways to poll the resultQ but for my purposes here this works pretty nicely. Anyway hope this is useful to someone...

Easiest Implementation *Ever* :)

So you say you want to perform a long running task from a wxPython GUI? Well, it don't get no simpler than this! Use threads but don't mess with a separate thread class. Make sure if you're running in the other thread that you interact with the GUI via wx.CallAfter.