charterm: Character-cell Terminal Interface

1Introduction

The charterm package provides a Racket interface for character-cell video
display terminals on Unix-like systems – such as for GNU Screen and tmux sessions on cloud servers, XTerm windows on a workstation desktop, and some older hardware
terminals (even the venerable DEC VT100). Currently, it implements a subset of features available on most
terminals.

This package could be used to implement a status/management console
for a Racket-based server process (perhaps run in GNU Screen or tmux on a server machine, to be detached and reattached from SSH
sessions), a lightweight user interface for a systems tool, a command-line
REPL, a text editor, creative retro uses of old equipment, and, perhaps most
importantly, a Rogue-like application.

The charterm package does not include any native code (such as from terminfo, termcap, curses, or ncurses) in the Racket process,
such as through the Racket FFI or C extensions, so there is less potential for
a problem involving native code to threaten the reliability or security of a
program. charterm is implemented in pure Racket code except for executing /bin/stty for some purposes. Specifically, /bin/stty at startup time and shutdown time, to set modes, and (for terminal
types that don’t seem to support a screen size report control sequence) when
getting screen size. Besides security and stability, lower dependence on
native code might also simplify porting to host platforms that don’t have those
native code facilities.

1.1Demo

For a demonstration, the following command, run from a terminal:

racket -em "(require charterm/demo)"

This demo reports what keys you pressed, while letting you edit a
text field, and while displaying a clock. The clock is updated roughly once
per second, and is not updated during heavy keyboard input, such as when typing
fast. The demo responds to changing terminal sizes, such as when an XTerm is
window is resized. It also displays the determined terminal size, and some
small tests of the #:width argument to charterm-display. Exit the demo by pressing the Esc key.

Note: Although this demo includes an editable text field, as proof
of concept, the current version of charterm does not provide editable text fields as reusable functionality.

keydec — how to decode key encodings of a particular
terminal. A keydec is constructed from one or more keysets, can produce keycodes or keyinfos.

keyset — a specification of encoding some of the keys in a
particular terminal, including keylabels and keycodes.

keylabel — a string for how a key is likely labeled on a
keyboard, such as the DEC VT100 PF1 key would have a keylabel "PF1" for a keycode'f1.

keycode — a value produced by a decoded key,
such as a character for normal printable keys, like #\a and #\space, a symbol for some recognized unprintable keys, like 'escape and 'f1, or possibly a number for unrecognized keys.

keyinfo — an object that is used like a keycode, except
bundles together a keycode and a keylabel, as well as alternatate keycodes and
information about how the key was decoded (e.g., from which keyset).

These terms are discussed in the following subsections.

charterm is developed with help of original documentation such as that
curated by Paul Williams at vt100.net, various commentary found on the Web, observed behavior with
modern software terminals like XTerm, various emulators for hardware terminals,
and sometimes original hardware terminals. Thanks to Mark Pearrow for
contributing a TeleVideo 950, and Paul McCabe for a Wyse S50 WinTerm.

At time of this writing, the author is looking to acquire a DEC
VT525, circa 1994, for ongoing testing.

The author welcomes feedback on useful improvements to charterm’s support for terminal diversity (no pun). If you have a terminal
that is sending an escape sequence not recognized by the demo, you can run the
demo with the -n (aka --no-escape) argument to see the exact byte sequence:

racket -pm- neil/charterm/demo -n

When -n is used, this will be indicated by the bottom-most scrolling line,
rather than saying “To quit, press Esc.” instead will say “There is no escape from this demo.” You will have to kill the process through some other means.

2.1Protocol

The first concept charterm has for distinguishing how to communicate with a terminal is what
is what is called here protocol, which concerns everything except how keyboard keys are decoded.
The following protocols are currently implemented:

ansi protocol — Terminals approximating [ANSI X3.64], which is most terminals in use today, including software ones
like XTerm. This protocol is the emphasis of this package; the other protocols
are for unusual situations.

wyse-wy50 protocol — Terminals compatible with the Wyse WY-50. This support is
based on [WY-50-QRG], [WY-60-UG], [wy60], and [PowerTerm]. Note that video attributes are not supported, due to the WY-50’s
model of having video attribute changes occupy character cells; you may wish
to run the Wyse terminal in an ANSI or VT100 mode.

televideo-925 protocol — Terminals compatible with the TeleVideo 925. This support is based on [TVI-925-IUG] and behavior of [PowerTerm]. Note that video attributes are not supported, due to the 925’s
model of having video attribute changes occupy character cells; you may wish to
run your TeleVideo terminal in ANSI or VT100 mode, if it has one.

ascii protocol — Terminals that support ASCII but not much else that we know about.

2.2Key Encoding

While most video display control, they seem to vary more by key
encoding.

The charterm author was motivated to increase the sophistication of its
keyboard handling after a series of revelations on the Sunday of the long
weekend in which charterm was initially written. The first was discovering that four of the
function keys that had been working fine in rxvt did not work in XTerm. Dave Gilbert somewhat demystified this by
pointing out that the original VT100 had only four function keys, which set
into motion an unfortunate series of bad decisions by various developers of
terminal software to be needlessly incompatible with each other. After
Googling, a horrifying 2005 Web post by Phil Gregory [Gregory], which showed that key encoding among XTerm variants was even
worse than one could ever fear. Even if one already knew how much subtleties
of old terminals varied (e.g., auto-newline behavior, whether an attribute
change consumed a space, etc.), this incompatibility in newer software was
surprising. Then, on a hunch, I tried the Linux Console on a Debian Squeeze
machine, which surely is ANSI, and found, however, that it generated yet different byte sequences, for the first five (not four) function keys. Then I compared all to the [ECMA-48] standard, which turns out to be nigh-inscrutable, so which might
help explain why everyone became so anti-social.

charterm now provides the abstractions of keysets and keydecs to deal with this diversity in a maintainable way.

2.2.1Keylabel

A keylabel is a Racket string for how a key is likely labeled on a particular
terminal’s keyboard. Different keyboards may have different keylabels for the
same keycode. For example, a VT100 has a PF1 key (keylabel "PF1", keycode 'f1), while many other keyboards would label the key F1 (keylabel "F1", keycode 'f1). The keylabel currently is most useful for documenting and
debugging, although it could later be used when giving instructions to the
user, such as knowing whether to tell the user the Return key or the Enter key; the Backspace or the Rubout key; etc.

2.2.2Keycode

A keycode is a value representing a key read from a terminal, which can be a
Racket character, symbol, or number. Keys corresponding to printable
characters have keycodes as Racket characters. Some keys corresponding to
special non-printable characters can have keycodes of Racket symbols, such as 'return, 'f1, 'up, etc.

procedure

(charterm-keycode?x)→boolean?

x:any/c

Predicate for whether or not x is a valid keycode.

2.2.3Keyinfo

A keyinfo represents a keycode for a key, a keylabel, and how it is encoded as bytes. It is represented in Racket as
a charterm-keyinfo object.

procedure

(charterm-keyinfo?x)→boolean?

x:any/c

Predicate for whether or not x is a charterm-keyinfo object.

procedure

(charterm-keyinfo-keyset-idki)→symbol?

ki:charterm-keyinfo?

(charterm-keyinfo-bytelangki)→string?

ki:charterm-keyinfo?

(charterm-keyinfo-bytelistki)→(listofbyte?)

ki:charterm-keyinfo?

(charterm-keyinfo-keylabelki)→string?

ki:charterm-keyinfo?

(charterm-keyinfo-keycodeki)→charterm-keycode?

ki:charterm-keyinfo?

(charterm-keyinfo-all-keycodeski)→(listofcharterm-keycode?)

ki:charterm-keyinfo?

Get information from a charterm-keyinfo object.

2.2.4Keyset

A keyset is a specification of keys on a particular keyboard, including their keylabel, encoding as bytes, and primary and alternate keycodes.

The means of constructing a keyset is currently internal to this package.

procedure

(charterm-keyset?x)→boolean?

x:any/c

Predicate for whether or not x is a keyset.

procedure

(charterm-keyset-idks)→symbol?

ks:charterm-keyset?

Get a symbol identifying the keyset.

value

charterm-ascii-keyset:charterm-keyset?

From the old [ASCII] standard. When defining a keydec, this is good to have as a final keyset, after the others.

value

charterm-dec-vt100-keyset:charterm-keyset?

From the DEC VT100. This currently defines the four function
keys (labeled on the keyboard, PF1 through PF4) as 'f1 through 'f4, and the arrow keys. [VT100-UG] and [PowerTerm] were used as references.

value

charterm-dec-vt220-keyset:charterm-keyset?

From the DEC VT220. This currently defines function keys F1 through F20.

From the current xterm-new, often called simply xterm, as developed by Thomas E. Dickey, and documented in [XTerm-ctlseqs]. Several also came from decompiling a terminfo entry. Thanks to Dickey for his emailed help.

value

charterm-rxvt-keyset:charterm-keyset?

From the rxvt terminal emulator. These come from [Gregory], and
currently define function keys 'f1 through 'f44.

value

charterm-wyse-wy50-keyset:charterm-keyset?

From the Wyse WY-50, based on [WY-50-QRG] and looking at photos of WY-50 keyboard, and tested in [wy60] and [PowerTerm]. The shifted function keys are provided as both 'shift-f1 through 'shift-16, and 'f17 through 'f31.

value

charterm-televideo-925-keyset:charterm-keyset?

From the TeleVideo 925, based on [TVI-925-IUG], [PowerTerm], and from looking at a TeleVideo 950 keyboard.

2.2.5Keydec

A keydec object is a key decoder for a specific variety of terminal, such
as for a specific termvar. A keydec is used to turn received key encodings from a terminal into keycode or keyinfo values. A keydec is constructed from a prioritized list of keyset objects, with earlier-listed keysets taking priority of
later-listed keysets when there is conflict between them as to how to decode a
particular byte sequence.

Keydec for any presumed ANSI-ish terminal, combining many ANSI-ish keysets.

value

charterm-insane-keydec:charterm-keydec?

Keydec for the uniquely desperate situation of wanting to possibly have
extensive key decoding for a terminal that might not even be ANSI, but be
Wyse, TeleVideo, or some other ASCII.

2.3Termvar

A termvar is what the charterm package calls the value of the Unix-like TERM environment variable. Each termvar has a default protocol and keydec. Note, however, that TERM is not always a precise indicator of the best protocol and keydec,
but by default we work with what we have.

3charterm Object

The charterm object captures the state of a session with a particular terminal.

A charterm object is also a synchronizable event, so it can be used with
procedures such as sync. As an event, it becomes ready when there is at least one byte
available for reading from the terminal, and its synchronization result is
itself.

This parameter provides the default charterm for most of the other procedures. It is usually set automatically by call-with-charterm, with-charterm, open-charterm, and close-charterm.

procedure

(open-charterm

[

#:ttytty

#:current?current?])

→

charterm?

tty:(or/c#fpath-string?)=#f

current?:boolean?=#t

Returns an open charterm object, by opening I/O ports on the terminal device at tty (or, if #f, file "/dev/tty"), and setting raw mode and disabling echo (via "/bin/stty"). If current? is true, the current-charterm parameter is also set to this object.

Note that this requires a Unix TTY to operate, so you normally
have to run it within a terminal window of some kind, not from within DrRacket.
Otherwise, you might get an error like:

open-charterm: stty "/dev/tty" failed

procedure

(close-charterm[#:chartermct])→void?

ct:charterm?=(current-charterm)

Closes ct by closing the I/O ports, and undoing open-charterm’s changes via "/bin/stty". If current-charterm is set to ct, then that parameter will be changed to #f for good measure. You might wish to use with-charterm instead of worrying about calling close-charterm directly.

Note: If you exit your Racket process without properly closing the charterm, your terminal may be left in a crazy state. You can fix it with
the command:

stty sane

syntax

(with-chartermexpr?...)

Opens a charterm and evaluates the body expressions in sequence with current-charterm set appropriately. When control jumps out of the body, in a
manner of speaking, the charterm is closed.

4Terminal Information

procedure

(charterm-screen-size[#:chartermct])

→

(or/c#fexact-nonnegative-integer?)

(or/c#fexact-nonnegative-integer?)

ct:charterm?=(current-charterm)

Attempts to get the screen size, in character columns and rows.
It may do this through a control sequence or through /bin/stty. If unable to get a value, then default of (80,24) is used.

The current behavior in this version of charterm is to adaptively try different methods of getting screen size,
and to remember what worked for the next time this procedure is called for ct. For terminals that are identified as screen by the TERM environment variable (e.g., terminal emulators like GNU Screen
and tmux), the current behavior is to not try the control sequence (which
causes a 1-second delay waiting for a terminal response that never arrives),
and to just use stty. For all other terminals, the control sequence is tried first, before trying stty. If neither the control sequence nor stty work, then neither method is tried again for ct, and instead the procedure always returns (#f, #f). This behavior very well might change in future versions of charterm, and the author welcomes feedback on which methods work with
which terminals.

5Display Control

5.1Cursor

procedure

(charterm-cursorxy[#:chartermct])→void?

x:exact-positive-integer?

y:exact-positive-integer?

ct:charterm?=(current-charterm)

Positions the cursor at column x, row y, with the upper-left character cell being (1, 1).

procedure

(charterm-newline[#:chartermct])→void?

ct:charterm?=(current-charterm)

Sends a newline to the terminal. This is typically a CR-LF
sequence.

5.2Displaying

procedure

(charterm-display

[

#:chartermct

#:widthwidth

#:padpad

#:truncatetruncate]

arg...)

→

void?

ct:charterm?=(current-charterm)

width:(or/c#fexact-positive-integer?)=#f

pad:(or/c'widthboolean?)='width

truncate:(or/c'widthboolean?)='width

arg:any/c

Displays each arg on the terminal, as if formatted by display, with the exception that unprintable or non-ASCII characters
might not be displayed. (The exact behavior of what is permitted is expected
to change in a later version of charterm, so avoid trying to send your own control sequences or using
newlines, making assumptions about non-ASCII characters, etc.)

If width is a number, then pad and truncate specify whether or not to pad with spaces or truncate the output, respectively, to width characters. When pad or width is 'width, that is a convenience meaning “true if, and only if, width is not #f.”

5.3Video Attributes

procedure

(charterm-normal[#:chartermct])→void?

ct:charterm?=(current-charterm)

(charterm-inverse[#:chartermct])→void?

ct:charterm?=(current-charterm)

(charterm-underline[#:chartermct])→void?

ct:charterm?=(current-charterm)

(charterm-blink[#:chartermct])→void?

ct:charterm?=(current-charterm)

(charterm-bold[#:chartermct])→void?

ct:charterm?=(current-charterm)

Sets the video attributes for subsequent writes to the terminal. In this version of charterm, each is mutually-exclusive, so, for example, setting bold clears inverse. Note that that video attributes are currently supported only for protocol 'ansi, due to limitations of the TeleVideo and Wyse models for
video attributes.

5.4Clearing

procedure

(charterm-clear-screen[#:chartermct])→void?

ct:charterm?=(current-charterm)

Clears the screen, including first setting the video attributes to
normal, and positioning the cursor at (1, 1).

procedure

(charterm-clear-line[#:chartermct])→void?

ct:charterm?=(current-charterm)

(charterm-clear-line-left[#:chartermct])→void?

ct:charterm?=(current-charterm)

(charterm-clear-line-right[#:chartermct])→void?

ct:charterm?=(current-charterm)

Clears text from the line with the cursor, or part of the line with the cursor.

5.5Line Insert and Delete

procedure

(charterm-insert-line[count#:chartermct])→void?

count:exact-positive-integer?=1

ct:charterm?=(current-charterm)

Inserts count blank lines at cursor. Note that not all terminals support
this.

procedure

(charterm-delete-line[count#:chartermct])→void?

count:exact-positive-integer?=1

ct:charterm?=(current-charterm)

Deletes count blank lines at cursor. Note that not all terminals support
this.

5.5.1Misc. Output

procedure

(charterm-bell[#:chartermct])→void?

ct:charterm?=(current-charterm)

Rings the terminal bell. This bell ringing might manifest as a
beep, a flash of the screen, or nothing.

6Keyboard Input

Normally you will get keyboard input using the charterm-read-key procedure.

procedure

(charterm-byte-ready?[#:chartermct])→boolean?

ct:charterm?=(current-charterm)

Returns true/false for whether at least one byte is ready for
reading (either in a buffer or on the port) from ct. Note that, since some keys are encoded as multiple bytes, just
because this procedure returns true doesn’t mean that charterm-read-key won’t block temporarily because it sees part of a potential
multiple-byte key encoding.

procedure

(charterm-read-key

[

#:chartermct

#:timeouttimeout])

→

(or#fchar?symbol?)

ct:charterm?=(current-charterm)

timeout:(or/c#fpositive?)=#f

Reads a key from ct, blocking indefinitely or until sometime after timeout seconds has been reached, if timeout is non-#f. If timeout is reached, #f is returned.

Many keys are returned as characters, especially ones that
correspond to printable characters. For example, the unshifted Q key is returned as character #\q. Some other keys are returned as symbols, such as 'return, 'escape, 'f1, 'shift-f12, 'right, and many others.

Since some keys are sent as ambiguous sequences, charterm-read-key employs separate timeouts internally, such as to disambuate
the Esc key (byte sequence 27) from what on some terminals would be
the F10 key (bytes sequence 27, 91, 50, 49, 126).

procedure

(charterm-read-keyinfo

[

#:chartermct

#:timeouttimeout])

→

charterm-keyinfo?

ct:charterm?=(current-charterm)

timeout:(or/c#fpositive?)=#f

Like charterm-read-keyinfo except instead of returning a keycode, it returns a keyinfo.

8Known Issues

Need to support ANSI alternate CSI for 8-bit terminals, even
before supporting 8-bit characters and multibyte.

Only supports ASCII characters. Adding UTF-8 support, for terminal emulators
that support it, would be nice.

Expose the character-decoding mini-language as a configurable
option. Perhaps wait until we implement timeout-based disambiguation at
arbitrary points in the the DFA rather than just at the top. Also, might be
better to resolve multi-byte characters first, in case that affects the
mini-language.

More controls for terminal features can be added.

Currently only implemented to work on Unix-like systems like
GNU/Linux.

Implement text input controls, either as part of this library or
another, using charterm-demo as a starting point.

9History

Version 4:5 — 2017-01-28

Documented about running within a terminal of some kind,
not DrRacket. (Thanks to Andrew Blinn.)

Version 4:4 — 2016-03-10

Added support for another stty-a format, to %charterm:screen-size-via-stty. (Thanks to Jay McCarthy.)

Added another variation of encoding for XTerm arrow,
Home, and End keys. (Thanks to Eli Barzilay.)

Version 2:1 — 2012-06-24

Corrected PLaneT version number in require in an example.

Version 2:0 — 2012-06-24

Greatly increased the sophistication of handling of terminal diversity.

Added the wyse-wy50 and televideo-950 [Correction: televideo-925] protocols, for supporting the native modes of Wyse and
TeleVideo terminals, respectively, and compatibles.

More support for different key encodings and termvars.

Demo is now in a separate file, mainly for convenience
in giving command lines that run it. This breaks a command line example
previously documented, so changed PLaneT major version, although the
previously-published example will need to have :1 added to it anyway.

charterm-screen-size now defaults to (80,24) when all else fails.

Documentation changes.

Version 1:1 — 2012-06-17

For screen and tmux, now gets screen size via stty. This resolves the sluggishness reported with screen. [Correction: In version 1:1, this behavior is
adaptive for all terminals, with the shortcut for termvarscreen that it doesn’t bother trying the control sequence.]

Documentation tweaks.

Version 1:0 — 2012-06-16

Initial version.

10Legal

Copyright 2012, 2013, 2016, 2017 Neil Van Dyke. This program is Free
Software; you can redistribute it and/or modify it under the terms of the GNU
Lesser General Public License as published by the Free Software Foundation;
either version 3 of the License, or (at your option) any later version. This
program is distributed in the hope that it will be useful, but without any
warranty; without even the implied warranty of merchantability or fitness for a
particular purpose. See http://www.gnu.org/licenses/ for details. For other
licenses and consulting, please contact the author.