Spynner is a stateful programmatic web browser module for Python.
It is based upon PyQT and WebKit.
It supports Javascript, AJAX, and every other technology that !WebKit is able to handle (Flash, SVG, …).
Spynner takes advantage of JQuery. a powerful Javascript library that makes the interaction with pages and event simulation really easy.

Using Spynner you would able to simulate a web browser with no GUI (though a browsing window can be opened for debugging purposes), so it may be used to implement crawlers or acceptance testing tools.

This method throws an exception on timeout, and can customize the default 30 seconds timeout.

But there, our target can randomly fails.
Instead, we will load and wait for something in the DOM to be there to continue.
We wait to have ‘aaa’ in the html, thus with unlimited tries at 1 seconds intervals each

>>> def wait_load(br):
... return 'aaa' in browser.html

Hit the wrong url, Eck, you are on an unlimited loop !:

# >> browser.load(bp+"html_controls.html", 1, wait_callback=wait_load)
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
content loaded, waiting for content to mach the callback
<Control-C>

Hit the wrong url, Eck, you are on an unlimited loop unless you wear condoms and set the tries!
It will throw an exception, but stop:

>>> ret = browser.load(bp+"/html_controls.html", 1, wait_callback=wait_load)
>>> [a for a in debug_stream.getvalue().splitlines() if 'SPYNNER waitload' in a][-1]
'SPYNNER waitload: The callback found what it was waiting for in its contents!'

Spynner uses jQuery to make Javascript interface easier.
By default, two modules are injected to every loaded page:

JQuery core Amongst other things, it adds the powerful JQuery selectors, which are used internally by some Spynner methods.
Of course you can also use jQuery when you inject your own code into a page.

[OBSOLETE, USE AT YOU OWN RISK, NO MAINTAINED, NO BUGFIX DONE] Simulate jQuery plugin: Makes it possible to simulate mouse and keyboard events (for now spynner uses it only in the _click_ action). Look up the library code to see which kind of events you can fire.

AS nowodays jquery is already included on major websites, so we must not inject if the javascript is already loaded by the targeted website.

You can parse the HTML of a webpage with your favorite parsing library eg: BeautifulSoup, lxml , or lxml, or …
Since we are already using Jquery for Javascript.
It feels just natural to work with pyquery, its Python counterpart: