The selection process is on the RT_GROUP_ICON, and then within the group on the ICON itself.

For the RT_GROUP_ICON process, modern Windows versions still use the algorithm used by Windows 95 (not that by Windows NT):

Windows NT simply chooses the first resource listed in the application’s RC script. On the other hand, Windows 95’s algorithm is to choose the alphabetically first named group icon if one exists. If one such group resource does not exist, Windows chooses the icon with the numerically lowest identifier.

Selecting the icon within a resource

When more images of that size are present, match on the best color depth for the display.

When no color depth matches, prefer the image with the greatest color depth not exceeding the color depth of the display.

When all exceed the color depth, prefer the lowest color depth.

For color depth, treat 8 or more bits per pixel as equal. So there is no advantage of including a 16×16 256-color image and a 16×16 16-color image in the same resource — the system will simply choose the first one it encounters. When the display is in 8-bpp mode, the system will choose a 16-color icon over a 256-color icon, and will display all icons using the system default palette.

I’m not completely confident how 32-bit precisely fits in this scheme. If someone knows, please let me know and I’ll include the information.