looking at the server log:
ERROR [org.rhq.enterprise.gui.definition.group.EditGroupDefinitionGeneralPropertiesUIBean] There was a problem calculating the results: javax.ejb.EJBTransaction...
WARN SQL Error: 1795, SQLState: 42000
ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-01795: maximum number of expressions in a list is 1000
this can easily be refactored to work with arbitrarily large result lists, but the priority probably isn't all that high. dynagroups have been out for 4 months now, and not a single customer has hit this issue - a good thing ; )

to test this, generate a large inventory. then use the expression:
groupby resource.type.plugin
calculate these results. if any of the resource groups that are created as a result have more than 1000 members, then this bug has been verified fixed.
if you don't have more than 1000 members in any groups, add more resources to your inventory, and then recalculate. repeat this step until you've achieved the needed size.

I populate a postgres db with 1000 additional tables, took it into inventory and recalculated an expression. Total number of members in the Postgres group was 1118. It recalculated all the groups without an error.
RHEL5.2, x86_64, PostgreSQL 8.1.11