How to do interactive console input?

My latest bjorne shell patch includes a first attempt at connecting standard input to the keyboard in a useful way. I've hacked the keyboard interpretter to map CTRL + key to control characters, and implemented a minimal TerminalInputStream that (just) maps ^D to a soft EOF marker. To see this working run:

$ chsh bjorne proclet$ cat > /jnode/tmp/foothe quick brown fox ...^D

You will immediately notice that the characters typed as 'cat's standard input don't get echoed, and that there is no line editting. This forum topic is about how to implement these.

In the good old UNIX world, echoing, line-editting, ^D, ^C, and lots of other terminal input-related stuff are implemented in the "tty" device driver stack, typically in the OS kernel. The behaviour of the terminal is controlled by a bunch of ioctls that set flags, specify active characters and so on. There is a lot of arcane stuff here, but the one big thing that this gives us is that a classic UNIX application can just read a stream of data characters without worrying about what is going on behind the scenes vis-a-vis line editing etc.

By contrast, the current JNode CommandShell deals with these issues directly, but only while the user is typing a command line. There is no support for connecting the keyboard to a command's standard input, or for a command opening "/dev/tty" - like stream. And these are hard for an application to implement "by hand" since they need to use the TextConsole layer to do character echoing, line editting, etc.

What I'm thinking is that we need to provide something that approximates the functionality of UNIX tty driver stack. This would be implemented as a filter stream that wraps a keyboard InputStream and a TextConsole. This would subsume the existing line editting, etc functionality in CommandShell.

Does this sound reasonable? Am I missing a simpler or better alternative?

The patch I just submitted does not address the following important issues:

How can we deliver properly encoded multibyte characters to an application's InputStream?

How can we deliver VK codes that don't map to Unicode codepoints?

(In addition, the patch cannot handle the NUL control character due to the unfortunate use of '0x0000' to represent an unmapped keycode.)

The KeyboardInputStream (as modified by the patch) drops the top 8 bits of character values in KeyEvents to make them 'fit' into the InputStream byte stream abstraction. KeyEvents that the keyboard interpretter didn't map to char values are filtered to get rid of modifier codes (SHIFT, CTRL, ALT, etc), and then the remaining VK codes are injected into the stream as bytes. This is broken both in theory and in practice. For a start, Latin-1 characters > 127 are liable to collide with significant VK codes.

I can think of a number of better ways to do this. Here are some examples:

Insert chars into the stream as UTF-8 sequences, and use some pre-existing scheme for encoding non-character VK keycodes; e.g. VT220 keyboard escape sequences. (This doesn't generalize. No existing scheme can deal with all possible current and future VK codes.)

Insert chars as above and use a JNode specific scheme for embedding unmapped VK codes. For example; use ESC xxxx {x=0..F} to represent the VK code xxxx, and ESC ESC to represent a real escape character. Or use the 0xffff Unicode 'not-a-character' codepoint as the VK code introducer.

Use an alternative to the InputStream API that distinguishes between character (Unicode) and non-character (unmapped VK) data points. For example, replace the read methods with different named methods for character and non-character data. We would need to do this in a way that doesn't involve breaking the InputStream API and the mountains of code that depends on it.

Once we've properly encoded the stream from the keyboard, we also need to be able to consume the stream in an appropriate fashion The TerminalInputStream class implements a rough approximation to the behavior of a UNIX tty stream in "cooked" mode. (Ideally the app would see properly encoded Unicode chars; currently, the app sees other 'crap' ... see above.)

We also need an approximation to a tty in "raw" mode, where line editting is bypassed, and an application can see both character and non-character VK codes.

A final piece of the input-side equation is that we will eventually need to cope with legacy terminal and xterm input. One idea is to implement alternative "keyboard interpretters" that translate all kinds of legacy codes (e.g. VT220 codes) into characters and VKs.

(We also need to enhance the output side as well; e.g. by emulating (say) a VT220's functionality for editting the displayed text in the JNode console stack. But that's a separate topic.)

I'm aware of the problem of line editting, it has poped up at the testing scripting language interpreters too, recently, but it's older than that.

Anyway I think we should handle two distinct cases with the commands.
One is the normal java behaviour, where the command or application gets the input characters on enter and up to that point the line can be editted at will. This would be the case of a standard java application starting up from the shell.
The other case is the jnode specific application which gets direct access to the stream of events and can takes actions on arbitrary keys or characters. This is the case of the shell or the charwa integration or the console base editors etc.

We need some refactorings to support these and I need some time to handle it.

Also extending support for command line history would be nice. So that for insatnce I start up the rhino shell or bsh and I have real line editting but also history of the entered lines on the arrow keys.

This is a sketch of what's on my mind related to this.
The bits of code for doing this are mostly there in the shell and console layers we need to reorganize and genearlize it a though.