Dialogs

Do not nest dialogs more than two levels deep. For example "Configuration dialog > Advanced dialog" is ok, "Configuration dialog > Advanced dialog > Further settings dialog" is too deep.

Avoid dialogs that contain only one or two options. If possible, use inline-editing instead. For example, to rename a list item, do not show a dialog but allow the user to change the label inside the list

Do not use dialogs to display non-critical messages which do not require any further user interaction (typically dialogs with a single "OK" or "Close" button). Use info panels instead. See Messages)

Position of dialogs

Always keep dialogs on top of their parent. This is usually taken care of by the window manager. But in composed applications, focus problems may occur. Check carefully in this case.

If reference in the parent window is required, make sure your dialog does not cover relevant parts. For example a find dialog should neither be centered, nor cover a search result.

Interaction

Set input focus on confirmation button.

Only use modal dialogs when:
Use modal dialogs only if allowing interaction with other parts of the application while the window is open could cause data loss or some other serious problem.
Provide a clear way of leaving the modal dialog, such as a Cancel button in an alert.