There is a conversation where Cygwin's maintainer (Corinna Vinschen) explains that the Cygwin pseudo TTYs look like pipes to the Microsoft Visual C run-time library (MSVCRT). She also suggests to implement a wrapper around the isatty() function that recognizes Cygwin pseudo TTYs.

The idea is to fetch the name of the pipe associated with given file descriptor. The NtQueryInformationFile function fetches FILE_NAME_INFORMATION structure, where FileName member contains the pipe name. If the pipe name matches the following pattern, then it is very likely that the command is running in interactive mode:

So the Cygwin part is already hacked. The next step is to make a Java function from the C code.

Example

The following creates ttyjni.TestApp class with istty() method implemented via the Java Native Interface (JNI). The code is tested on GNU/Linux (x86_64) and Cygwin on Windows 7 (64-bit). The code can be easily ported to Windows (cmd.exe), maybe even works as is.