I'm making a program to save certain keyboard or mouse actions you have done on your computer and reproduce them for you, helping out with a lot of things like mundane multimedia tasks and whatnot. I want you to be able to provide conditionals that will cause the program to produce certain actions, and these conditionals I want to rely solely on what is being displayed on your monitor.

Like, for example, I want the program to automatically log in to something for me once it opens, like JGO. It would need to scan the screen to see if it perhaps had the JGO banner in the expected place. If it did, it would automatically click in the login box, type out my name, then the password box and type out my password.

Yes, pretty ambitious, but I've got pretty much everything working already. I'm just wondering if there's some way to return what's being displayed as a BufferedImage, or to get a screenshot of it, at least, which may be a little processor intensive but it would certainly work.

for me I get about 14 screen captures per second on a 1024x768 screen. if you already have a presumed position for, say, the JGO banner, you can just use a smaller Rectangle for the screen capture and it should be alot faster than taking an ENTIRE screenshot

Ooh yeah, that's working really fast. I just have the user specify a certain rectangle that they want to search for, generally not larger than 50x50 pixels. 50x50 pixels goes really really fast. I can also put in the option to scan the screen for certain patterns, and therefore it would work across multiple resolutions, at least somewhat (look for the text box pattern, because it will always be the same size, etc.)

I wonder if there would be any way to get this to work flawlessly aside from forcing a resolution change. The windows will obviously move around and be in different places depending on resolution, even depending on computer. I suppose I could do something called findWindow() function that finds the window in question so it is only performed once to get a relative position, then smaller screen analysis can be performed. But can you think of another way to do this?

Yeah, this is wholly true, but even if I could contact Windows does that mean that I could have Internet Explorer return to me its window size, font size, locations of text fields, etc.? I feel like even if I could do that it would just because program specific. I want this to be functionable through any program as long as the user saves a set of actions depending on different highlights on their screen.

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org