On Linux and Mac OS X, is there a kill or killall command that can let the app ask for unsaved document before exiting, kind of like asking the app to quit, or clicking on the "close" button of the window.

For example, type somecommand TextEdit on Mac and TextEdit will ask you if you want to save the unsaved document.

Signals that terminate an application aren't meant to allow the application any further user interaction. Either the user is no longer available (the HUP signal, whose original meaning was that the user was connected to the computer through a modem and the phone died), or the user doesn't want to be bothered (the default TERM signal and its more violent siblings QUIT and KILL). Many applications will try to save unsaved work somewhere if they receive the HUP signal, though.

A different option for Linux (and other unices) is to send a notification to the window, rather than the process: the _NET_CLOSE_WINDOW message tells an application to get rid of a specific window gracefully. You can send this message from the command line with wmctrl, e.g., wmctrl -c WINDOW_TITLE_SUBSTRING. Unfortunately, not all applications support this.