Try this: Take 1146 REALbasic projects and build them. Write down a log which files worked and which failed.
That is the job I did today. I have a tool to help me as I need to verify whether my example projects do compile. That does not tell me whether they actually still do their job right, but it catches syntax changes in the plugin or functions which are changed or removed. So please tell me if an example does no longer work.

What do you think you need to get those projects tested?
First you need an older REALbasic version. 2009r2 is okay for this. Why? Because REALsoftware changed editfield to textarea and textfield and if you load a project with editfields in a newer RB version, you get a dialog and your automation stops. Unless you do something like magical keystokes. But that would not allow the whole process to run in the background. By the way I want to keep the editfields in the examples as my examples should be compatible with RB 2008 to 2010.

Okay, next you would want to learn the REALbasic file format. Why? Well, because RB complains if it opens a file of a future version. With a thousand files, you don't want to make manually sure that all RB files are old enough. So you change the version in the beginning of the file.

Next you start with the IDE Scripting stuff from REALsoftware to execute scripts from the shell in RB. Basicly you launch a project in RB so it loads. Than you send a build command over the IDECommunicator module. You certainly want your script to use a shell command to write a file with success or failure to the disc. Than you can issue a command to close the project.

Still you have a problem. Every x builds REALbasic crashes. For me restarting RB every 50 projects works quite fine. I don't know the exact number, but at some point the memory is full or some other internal tables so it simply crashes.

So if you know something or work on REALsoftware, you can help me with ideas for this items:

Keep RB away from crashing after x projects

Find a way to disable the version warning dialog

Find a way to disable the deprecation warning dialog with the editfield/textfield complains