Posted
by
kdawsonon Tuesday April 06, 2010 @08:16AM
from the latter-day-screen-scrape dept.

The Installer writes "Researchers from the University of Washington have managed to add customization and accessibility options to proprietary software without ever touching the source code. Rather than alter program code, Prefab looks for the pixels associated with the blocks of code used to paint applications to a screen, grabs hold of them, and alters them according to whatever enhancements the user has chosen to apply. Any user input is then fed back to the original software, still running behind the enhanced interface."

'Closed software' is a fact of life for most users. This attempt at 'expanding' the functionality isn't very impressive, though, and won't have very many real world uses. What if you resize your monitor, do your 'customizations' all go to hell?

I always liked using the plugin architecture for applications that provide it.

I can see many uses for this - but then, I've got AutoIt for the sorts of things I'm thinking of already.

I think the article title and summary are completely bunk, though. They're not making the software any more -open- than it was before; If you have a button that writes "Hello World" to a file, then you can replace that button with a contrast-rich enlarged version with excellent text-to-speech functionality, or make it a bouncing spinny glowing orb like something out of Kai Krause's mind... but pressing

sometimes those custom UI elements simply are more appropriate than the widgets that come with the OS/widget provider.

but then sometimes they are simply included for no reason in a program in order to make it bigger. When was the last time you bought a printer or scanner (or both) for a Windows machine? I'll bet you anything that the bundled software came with some jinkety-ass homescript "hardware manager" made by HP or Canon or whatever, with a 45MB executable binary, using a statically linked UI toolkit w

What if you resize your monitor, do your 'customizations' all go to hell?

How often does that happen, though? Personally, I've used laptops pretty much exclusively for the last six or seven years. Which means resizing the monitor means getting an entirely new system, something I only do once every two years or so. I'm more than happy to go through the steps to get this kind of thing working once every two years if it means I can customize software I have to use for work to make it easier for me to do my job.

Watch the youtube video, they have the software running on Vista work on widgets drawn in a OSX remote desktop, this can totally handle you moving windows around/changing screen resolution. I would be more worried if it can work on loads of different Gnome GTK themes, as the widgets will be less consistent.

Closed software' is a fact of life for most users. This attempt at 'expanding' the functionality isn't very impressive, though, and won't have very many real world uses. What if you resize your monitor, do your 'customizations' all go to hell?

Not to mention the legal issues. Or trying to keep up with changes intentionally made to break your efforts (just ask the WINE, SAMBA or iPod-Linux compatibility devs about this).

The first time I saw this article in ACM links I thought "neat, but what a waste of effor

I enjoy customizing my apps by sub-classing windows or whatever else is needed.

I fixed a particularly stupid application for work which requires you to click on a calendar and choose a date, and again on another calendar. Clicking OK updates the data source, but clears all the values. So every day when you're going through today's data, you select both calendars and click OK and re-set both calendars. Lots of wasted time. I updated the button click to save the calendar date, call the normal click functi

It's not expanding the functionality. It's modifying the GUI.Maybe adding an option checkbox that wasn't there, or rearranging some dialog somewhere.

Without the source code, and more specifically, modifying the source code, it's impossible to add features in any meaningful sense.

Sure, you could add a "show all images with red background" checkbox, or something like that, and modify the program display output appropriately, but I can't imagine any case where something like that would be called a feature.

It is most probably breaking the EULA.
And experience has taught me to be weary of what seems rational, logical and full of common sense in the legal world...

When I read such things on the project made to allow OS X to run on a PC:http://wiki.osx86project.org/wiki/index.php/FAQ#Legal [osx86project.org]
I can only think that someone will complain sooner or later. I can't see Adobe accepting the sale of something customizing paintshop pro without getting some bucks from it.

I have to imagine that, if this software is intercepting the outputs of legally-paid-for closed source software and altering them, this could never be considered a crack. Then again, I suppose if Hollyweird can sue someone for building a custom version of a movie with the swearing and naughty bits bleeped out, while including a copy of the original version of the movie to make sure the end consumer has actually purchased a license, who knows?

If this is considered a "crack", will software developers be able to stop me from purchasing a larger screen, or better speakers?

Then again, I suppose if Hollyweird can sue someone for building a custom version of a movie with the swearing and naughty bits bleeped out..

Actually that bugs the hell out of me. If a movie comes on TV I want the see the movie as intended, not some hacked up derivative based on someone elses misdirected moral values. I should decide what I can and cannot watch - not someone else. For the most part I've stopped watching movies broadcast on US television because they are so hacked to pieces they are unwatchable. Usually in Canada a movie on TV will have warnings on it stating what the content is, but not be edited.

But the company in question packaged the original movie along with a "sanitized" version. You get to decide what you can and cannot watch, by either pulling the original or the "sanitized" version from the DVD holder.

Silly? Sure. If I don't like a movie in its original, I probably won't like a sanitized version either.

But it's not affecting your choice to watch movies the way you want to - it's offering a service that is optional and people pay extra for.

But the company in question packaged the original movie along with a "sanitized" version.

So what? How does that change the fact that they still need permission to create the sanitized version? Such a work wouldn't fall under fair use. This seems to be the same bullshit logic that people used to claim about downloading game ROMs. You don't just get to download ROMs off the internet just because you may own the original game.

Then again, I suppose if Hollyweird can sue someone for building a custom version of a movie with the swearing and naughty bits bleeped out, while including a copy of the original version of the movie to make sure the end consumer has actually purchased a license, who knows?

Except just because you have a license to the movie doesn't mean you get to do with it as you please. Fair Use wouldn't cover creating a whole new version of the movie with parts cut out and swearing bleeped.

While the power of the almighty EULA and(in the case of software with embedded DRM/anti-tampering features, the DMCA) might well cloud the issue, there is an analogy worth looking at.

A while back, there was a company called "CleanFlicks" that operated a movie rental service, aimed mostly at Fundie Mormons and the like. They took DVDs, reviewed them, and produced bowdlerized versions, which they then rented out. They were sued, and lost, on the grounds that they were, indeed, producing and trafficking in unlicenced derivative works.(IRRC, they purchased as many DVDs as they rented out, so they weren't illegally duplicating, ie. 1 purchase to 1 edited rental copy, in any useful way; but they were still smacked down). They exist today with a much reduced catalog of movies that fit their standards without editing.

A similar company "ClearPlay" used a different tactic. They provided specially programmed DVD players that were able to interpret a control file(programmmatic mute/unmute, FF/play, etc.) and rented unedited commercial DVDs, along with matching control files that, when used with their players, automatically "edited" the DVD as you watched it. The MPAA threw a fit; but the company survived legal challenge(it helped that congress, tipping their hat to "the children" passed a law to explicitly clarify the scope of copyright on this point). Since, unlike "CleanFlicks", they weren't actually creating a derivative work, just a control file that modified the behavior of the DVD player during playback, they were judged to be in a different category.

Again, barring the sorts of tricks that can be pulled with even weak DRM+DMCA, this sort of "customization kit" tech would probably fall into the "ClearPlay" side of the analogy. Actually selling edited binaries, even if you purchased a legitimate copy for each edited one you sold, would almost certainly put you in the "CleanFlicks" camp, and get you smacked down; but selling a customization package that modifies the appearance of a binary only at the point of execution on the end user's computer, or even selling a bundle of "copy of commercial software + installer for customization kit" would probably pass legal muster.

The only complication, of course, is that court decisions are, in practice, driven by a mixture of the text of the law and a somewhat emotive sense of "intent" or "desirable outcome". Protecting the kiddies from corruption generally wins you warm and fuzzies. It isn't clear that modifying the appearance of dialog boxes would have the same cultural clout(unless you could, say, find a nice test case involving a bunch of blind kids who are able to use $SOFWARE_X with their screenreaders for the very first time*wipes tear* or something of that sort).

Oops, forgot to mention: Any precedent established by cases surrounding "Game Genie", "Gameshark" and the various other cartridge pass-through modding systems might be even more relevant.

Again, the DMCA can be used in some rather nasty ways; but my understanding (IANAL) is that "Game Genie" and friends, while strongly disliked by the console makers, generally survived legal challenge, because they were pretty clearly only useful for letting people who had purchased cartridges make fair use of them on the

How about skipping the nag screen of a freeware ?
To take a real example, last time I clicked on a.zip file on a vista computer, winzip opened and started to count slowly the number of days it was used without being bought. I have the feeling that if I were to bypass this, a court could feel I am circumventing something.

(it helped that congress, tipping their hat to "the children" passed a law to explicitly clarify the scope of copyright on this point)

If they had to do this, it shows that the matter is not evident with regard to the law.

How about skipping the nag screen of a freeware ?
To take a real example, last time I clicked on a.zip file on a vista computer, winzip opened and started to count slowly the number of days it was used without being bought. I have the feeling that if I were to bypass this, a court could feel I am circumventing something.

That's the sort of thing that I was thinking of when I said 'The only complication, of course, is that court decisions are, in practice, driven by a mixture of the text of the law and a somewhat emotive sense of "intent" or "desirable outcome".' Unless copyright law is particularly clear on what exactly "derivative work" means(and, given that programmatic graphical screen-scraping and realtime modification of a program's apparent user interface without actually modifying the binary almost certainly wasn't e

No we aren’t. But many (pretty dumb) people have the perception that we would, because they think that others would have that perception, because (1:) they said something alike, because they themselves think that others would have that perception, because... GOTO (1).

Or in other words: Monkey, see, monkey think, monkey parrot. ^^

What most people don’t know, is that it’s all in our head. Like fashion. Why did women think that wearing rubber boots with colorful flowers on them in the middle of the summer would be cool? Because “it was the fashion at the time“. And why was it that? Because some self-appointed “fashion expert” told them so. (Because he sold that stuff. But that’s another story.)

But I think in this case there wasn’t even really a starting point. There was some perception that “that’s how it is”, because the closed source people acted more secure, because, being business people, they were trained that way.

The thing is: Why would you let it just grow on us like a mind-virus, when you can change it just like that? After all, if it’s all in our heads, it only requires some of us to always securely act how it really is: We are gaining, winning, and rolling over them in an unstoppable wave!Have you ever noticed how a big mass of people switched their mind sets, and started to become a raging mob, or something like that? It’s that exact thing. It only needs a seed (you) who is so secure, that they start to doubt themselves. Then the rest is only a matter of that self-amplifying mechanism above, and time (depending on how strong your reality is).

It’s a big war of psychology and social engineering, for the minds of people. And I don’t let the politicians, mass media or greedy multinational corporations win it.:)Think like this: You’re a mind hacker. And the mind is by far the most complex computer ever. Isn’t that much cooler than doing it to such a (in comparison) ridiculously simple thing as a computer (network)?

If you RTFA or WTFV, you'd know that it's detecting the input elements using an algorithm and not hard coded to the specific application (they even demoed VNCing into an OS X machine and having it detect the UI elements there and applying the processing).

And if you've worked with screen scrapers, you'd know that most of them are based on UI elements, and an upgrade to the underlying software almost always causes problems because the UI elements frequently change when software is upgraded.

So, yes, this is a screen scraper, which means it will survive some small alterations to the UI, but you're usually looking at upgrading/rewriting the screen specs when the underlying screens change.

And if you've worked with screen scrapers, you'd know that most of them are based on UI elements, and an upgrade to the underlying software almost always causes problems because the UI elements frequently change when software is upgraded.

Screen scrapers fail when the UI is updated because they need to be able to find and scrape specific bits of data. They need to know which widgets map to which values. By contrast, this software only cares about locating standard widgets on the screen. End of story. It doesn't care what values are represented by those widgets, because that's not relevant to its functionality. So it should work just fine for any application that uses a known set of widgets, since all it needs to do is be able to say, "A

So when I add a checkbox that says "Amortize this loan?" above the checkbox that used to be there that said "Automatically calculate payments for this loan?" then change the verbiage on the automatic calculation, this application is going to go kaboom.

I'm not criticizing screen scrapers. I've used them. But upgrading the underlying software always changes screen elements and requires a rewrite/update to the scraper code.

So, in answer to the original question:

This is called a screen scraper, and likes to break with updates to the underlying program, right?

So when I add a checkbox that says "Amortize this loan?" above the checkbox that used to be there that said "Automatically calculate payments for this loan?" then change the verbiage on the automatic calculation, this application is going to go kaboom.

I think we're talking about two different things, here. The application that was demonstrated in the linked article is not a screen scraper in the traditional sense. It doesn't read any values from the widgets. It doesn't care what the widgets represent. All it does it recognize that the widgets exist, and causes the cursor to either slow down over, or snap to, widgets.

So moving widgets around doesn't faze it, because it was never differentiating between them to begin with. All it cares about is loc

This isn't just about recognising GUI elements just to slow down but also to present a partiall or wholly alternate interface to the user to interact with.

Whatever markup they are using after element detection would have to map the original gui element to some new gui element(s) possibly elsewhere on the new interface, possibly with new text/mouse over behaviour/etc.

So, a simple app with two buttons, "Quit" and "Do Something" may have a Gui wrapper that mak

Open Source as a development methodology has already won. It is more scalable: more features, custom features, localization, error removing all work better with it. It is also a lateral organization like a web which avoids some issues that the "Mythical Man Month" talks about which results from hierarchal organization. Close software will never go away but it's utility has already been greatly compromised by Open software. With Open software this extra translation layer is unnecessary you would just modify the source and have a more reliable program intrinsically. Code is Free, get over it.

Open source may technically be a better development model for those reasons, but you need to have interested developers to work on your projects, and the only way to get people interested in some projects is simply to pay them. It won't matter how open or closed the source is if nobody is interested in helping out. So closed source is just as good if not better than open source for that type of scenario, because it will get the hours of work it needs put in there. Even if a closed source equivalent does bec

That has to add a lot of overhead to the already running process and to what benefit? If it's reading the code "as many as 20 times per second" that is going to add tons of CPU and RAM usage to the system that just isn't needed. F/OSS ftw!

hat has to add a lot of overhead to the already running process and to what benefit? If it's reading the code "as many as 20 times per second" that is going to add tons of CPU and RAM usage to the system that just isn't needed. F/OSS ftw!

With i7 chips, SSD HDs, 64bit OSes that support 4+ gig of triple chan memory (any or all of those in one machine are huge improvements in desktop computing power) you'll still not push it to capacity with 20 such apps running. We are at a point where we have an abundance of CPU/memory to spare, I see nothing wrong with developing such apps (if only as stop gaps) until such time that a suitable replacement arrives. These apps very well may be the impetus for the development of those open apps once it prove

This sounds like the same sort of attitude that software writers have had for ages. Just write bloated, inefficient code and let processing power eventually catch up to run the software. I think that this needs to be a legitimate concern or we will wind up back to the point where a new version of Windows would come out and barely be able to run on the technology available. Yes, this may not add much to a computer running a Core i7 with 6 gigs of memory, but that sort of system is pretty rare in the real

My mom is running ~.5GB RAM on her XP system and it's not nearly enough. What I have is barely enough. To have another app laying up in ram running through each instruction of any program I opened at 20 times/second...no. That's not happening on either system.

> If it's reading the code "as many as 20 times per second" that is going to add tons of CPU and RAM usage

Why? 20 times per second is nothing. Our PCs can do billions of operations per second. I doubt a few basic UI tweaks will make even a 1% difference to CPU loading unless it's very badly written. The way it works isn't even that different to how skinnable applications work natively. Sure there's an extra step - but those steps will only a take a couple of milliseconds out of every second.

There is an exponential difference between "20 times per second" and "20 instructions per second". I don't know how much headroom this would generate, but I know it would likely be a few million times more than "20 instructions per second".

CPU is cheap, but reverse engineering pixel clusters to identify
buttons, text boxes, toolbars etc is expensive in programmer time, and
each time the underlying UI/theme changes, it needs to be redone. The
same is true if you run a variety of apps with their own custom UI
elements/themes, which is pretty common on Windows (even single
vendors like Microsoft aren't consistent in UI toolkits, eg Office
apps don't share the same widgets as Notepad).

How many cubicle farms are going to put high end graphics cards on their workstations? If I were a responsible IT Manager, I would only allow the minimum needed to run the OS (office apps and most specialty software should not need more than that). From a cost perspective, that would be responsible. Running a high end graphics card just to assist in running this monstrosity would be irresponsible at best.

I remember doing hacks like this to Mac applications -- back when they had resource + data forks. The resource forks contained all their sounds, graphics, icons, forms, etc. With ResEdit you could simply open up (most) applications and tweak them to your hearts content:)

Applications don't really use them anymore, except to store icons. OS X applications are folders named with a.app extension, which the OS abstracts to a single icon. The reality is that the executable is just one file inside this folder somewhere. None of those files are in the resource fork anymore.

From the summary I thought this sounded more analogous to writing a GUI front end for a command line program rather than simply changing the graphics.. you add another layer on top that takes a high level command then performs the intended action in the application, but with less needless tedium.

I remember doing hacks like this to Mac applications -- back when they had resource + data forks. The resource forks contained all their sounds, graphics, icons, forms, etc. With ResEdit you could simply open up (most) applications and tweak them to your hearts content:)

These days, if you want to customize an OS X app, you just open up the.app folder, and play with the.nib files. It's actually kind of fun to open up the.nib's in Interface Builder and change around all the buttons for iTunes or whatever

You can add "features" in terms of "laying things out slightly differently, or providing highlights and other feedback, all of which still interfaces with an identical feature set of the app underneath".

Or "it is acting like it is open in the same way as straping stabilisers to a motorbike makes it like a sports car".

Chances are that such "enhancements" constitute a derivative work of the proprietary user interface. In many jurisdiction even using such a thing is illegal and distributing the enhancements without permission from the copyright owner most certainly is. Which, of course, is very different from free and open source software, where people are encouraged to share and improve the code.

It's only a derivative work if you distribute the proprietary software along with your enhancement. If the enhancement simply requires that the user already have a copy of the proprietary software in order to use it, then it's not a derivative work.

It's only a derivative work if you distribute the proprietary software along with your enhancement. If the enhancement simply requires that the user already have a copy of the proprietary software in order to use it, then it's not a derivative work.

Depends. If the enhancement has significant chunks of the original app's UI included within it (e.g. for purposes of recognising specific dialogs, etc.), it could be considered derivative. This is the same theory as states that an app compiled against a library,

Chances are that such "enhancements" constitute a derivative work of the proprietary user interface.

No, it isn't. If I buy ten copies of LOTR and write in the margins and resell those books, I have not created a "derivative work". If I reprint those ten books with the extra notes, even if I include the original copies, then I have created a derivative work and have infringed copyright.

No way is changing the output of a program a "derivative work"; it's writing in the margins, and not even reselling the book

and distributing the enhancements without permission from the copyright owner most certainly is.

Now ya got it!

Just to be clear: Restrictions around derivative works involve *distributing* those works. But if an individual uses a tool like this to create a "derivative work" for his own personal use, that doesn't run afoul of the law. So, no, simply *using* this tool is not illegal. But if you were to take the application, wrap it in a too

Tools like this have been around for ages, although they are usually called "GUI test frameworks" or "automation assistants". Such tools provide a way of scripting interactions with an existing GUI. However, they really only focus on mimicking pre-recorded user interactions--it seems like it would be quite time-consuming and fragile to reverse-engineer an arbitrary program's dialogue boxes in a robust way that would allow control to be significantly enhanced.

Also, on Windows, at least, there are tools that enhance the operation of dialogue boxes (for example, adding history and options to the File Open dialog). Those tools work at a more abstract level than snagging pixels, which is a lot more efficient--but that means they are ineffective on applications that have already customized those dialogues.

It seems like the fundamental non-breakthrough here is that the application actually must already include the functionality that you want to express in your modified UI--otherwise, you can modify the UI all you want, but the app will only do what it's capable of. So if you want it to display a bunch of different renderings in sub-windows automatically (to use their example), it had better be capable of that display already.

Tools like this have been around for ages, although they are usually called "GUI test frameworks" or "automation assistants".

Tools have been around for ages that present a new GUI while hiding the old one and proxying events from the new interface back to the old one, so that you can retool the UI without modifying the application?

They hijack your client display when you access a banking page, fill in the account details and amount for you, while presenting you with interface to enter your target account, then replace the data on the confirmation page with whatever you entered while obscuring the data they entered. You sign the transaction with a token or OTP and instead of sending $15 to your aunt, $10k money gets transferred to the hijacker's account in Nigeria.

I did this years ago back in 1998 when I had an issue getting my MAPI service provider working to provide an interface to my companies calendaring tool. I hijacked the Outlook calendar to create items in my software, while supporting regular Outlook email. Good stuff.

There was a shareware Windows program called "The Customizer" which recognized a parent window, then would move, resize, hide/show, enable/disable all the child windows (buttons, controls, etc) the way you want it every time that window appears. So you could completely rearrange a dialog box of a program, turn on some hidden controls, enable stuff that's supposed to be disabled, etc.

Then there's also resource file editing, you can change dialogs, or add shortcut keys to menu items or other dialog buttons.

I'm waiting for a game 'bot which uses a camera to look at the screen. That would work on consoles, even ones that used HDCP displays. The future of gold farming is
a dimly lit room with hundreds of shelves of game consoles, and cameras pointed at small screens, tied to a farm of 1U servers playing the games.

Is this article essentially describing a way to re-open the analog hole that HDMI is looking to close? I mean, if you could show your Blu-Ray movie over HDMI, then capture the pixels and save them to your hard drive, haven't you just opened a new hole for people to record content?

Greasemonkey + AdBlock Plus + NoScript = The Web, the way YOU want it to be.:)

And, just like the tool above, if a company changes their web page, you're looking at some redo on at least the Greasemonkey site. Be sure to add Greasefire in addition to Greasemonkey - lots of people have lots of great scripts that are at least good example code to pull from.