Usual ways to find number of open windows in a process? (reverse engineering)

If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register or Login
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Usual ways to find number of open windows in a process? (reverse engineering)

I'm trying to reverse engineer a program that sees how many windows it has opened. The max number of windows it allows are 6, and I want to change that limit. I can find the address that stores the value on open windows, and lock it (at 1 for example, so the program would think it only has 1 open window), but the program still detects when it reaches the limit of 6 windows.
So I guess that when it opens a new window, it doesn't really check that variable, and computes on-the-fly how many windows are open. So I put a break-point at calls of the functions EnumWindows(), EnumChildWindows(), EnumThreadWindows(), FindWindowA(), FindWindowW(), FindWindowExA(). Still, the program detects when it reaches the 6 open windows. Am I missing any function? How is the program seeing how many open windows it has?

Re: Usual ways to find number of open windows in a process? (reverse engineering)

Originally Posted by Toshioo

I'm trying to reverse engineer a program that sees how many windows it has opened. The max number of windows it allows are 6, and I want to change that limit. I can find the address that stores the value on open windows, and lock it (at 1 for example, so the program would think it only has 1 open window), but the program still detects when it reaches the limit of 6 windows.

Well, does this program (or who has rights to this program) allow you to perform such a "reverse engineer"?
If not - see the Acceptable Use Policy ("AUP"). For example:

You will not use these Posting Features for the purposes of sharing or distributing viruses, licenses, registration information, software keys, "cracks," or other information designed to do harm to or allow unlawful access to any computer hardware, software, networks, or any other systems.

...
Except where expressly permitted, you do not have the right to use, copy, transmit, modify or repurpose any Content that does not belong to you which is on the Websites or made available through the Posting Features. At all times, you remain solely responsible for anything you post and agree to indemnify and hold WebMediaBrands and its subsidiaries, affiliates, officers, directors, agents, sponsors, advertisers, partners, and employees, harmless from any claim or demand, including reasonable attorneys' fees, made by any third party due to or arising out of any material you submit, post to or transmit through the Posting Features, your use of the Posting Features, your connection to the Posting Features or Websites, your violation of the AUP, Legal Notice, or Privacy Policy, or your violation of the rights of another.

Originally Posted by Toshioo

... I put a break-point at calls of the functions EnumWindows(), EnumChildWindows(), EnumThreadWindows(), FindWindowA(), FindWindowW(), FindWindowExA(). Still, the program detects when it reaches the 6 open windows. Am I missing any function? How is the program seeing how many open windows it has?

Do you have a source code do debug this program? Or how else do you expect the program will break at one of those functions?

Re: Usual ways to find number of open windows in a process? (reverse engineering)

I think this approach is very naive. Why would I use any of Enum...() or Find...() functions to locate the window that I created? I would much rather keep a handle in some container.
The program you are talking about might have a fixed-size array of handles, and if you trick it into creating one more window, it might write out-of-bounds causing memory corruption or access violation.
Are you sure there is no configuration parameter to change the max number of windows?
Could you just open another instance of that program to get six more windows?