Matplotlib's plots actually have some nice event handler capabilities which make it very easy to extend a view with additional mouse-keyboard driven functionality (very conveniently, events are reported with both view and data coordinates). I've certainly used it to add mouse-drag driven contrast-brightness adjustment to a displayed image, and to launch algorithms needing an initial seedpoint.