z-index is a css property that applies to HTML elements only, thus the modeless dialog window (or any window object) will not have the property available.

Unfortunately there are no properties or methods or the layering/ordering of windows in javascript. At best you would probably need to come up with a workaround where you make sure you keep up with each new window object in the parent window (each one needs it's own variable). You could store the newly created window object along with an 'order' in your own object and attempt to keep track of the order via pushing data from each child window back to the parent (for instance when a window is focused, gaining the top index it pushes this back to the parent, which updates your object).

There would be more work needed to properly manage indexes, such as the child windows properly updating this variable in the parent when they are focused. But again this is just an example of a workaround since there are no actual ways of getting the index/order of windows created in javascript (modeless or otherwise).

Now with all of that being said to directly address your question I do have to ask, why use showModelessDialog() in this situation? It's an IE only function, thus compatibility is very limited. But even more so, for your specific case I feel that you should try using inline windows (eg. jQuery modal windows). To be fair I haven't messed with multiple jQuery modal windows at a time (or the layering/order/index of multiple windows) but even without jQuery you can still use a similar method of creating HTML elements that 'act' like windows and allow you to directly control the order of them as they are all child elements of that document.