Type-safeness in Shell

Since writ­ing the post on a hy­po­thet­i­cal hull lan­guage as an al­ter­na­tive to shell I can­not stop think­ing about the short­com­ings of shell.

And one think that comes to mind over and over is type-safe­ness. Shell treats ev­ery­thing as a string and that’s the source of both its power and its poor main­tain­abil­ity.

So when I ask whether shell can be im­proved, the ques­tion is ac­tu­ally more sub­tle: Can it be im­proved with­out com­pro­mis­ing its ver­sa­tility? Can we, for ex­am­ple, be more type-safe with­out hav­ing to type Java-like stuff on the com­mand line? Without sac­ri­fic­ing the pow­er­ful and dan­ger­ous fea­tures like string ex­pan­sion?

I mean, you can write shell-like scripts in Python even to­day and use type hints to get type safe­ness. But in real world this prac­tice seems to be re­stricted to writ­ing more com­plex pro­grams, pro­grams that re­quire ac­tual in-lan­guage pro­cess­ing, com­plex con­trol flow, use of libraries and so on. Your typ­i­cal shell script which just chains to­gether a hand­ful of UNIX util­ities — no, I don’t see that hap­pen­ing a lot.

To put it in other words, differ­ent “script­ing lan­guages” man­aged to carve their own prob­lem spaces from what once used to be the do­main of shell, but al­most none of them at­tacked its very core use case, the place where it acts as a dumb glue be­tween stand-alone ap­pli­ca­tions.

But when writ­ing shell scripts, I ob­serve that I do have a type sys­tem in mind. When I type “ls” I know that an ar­gu­ment of type “path” should fol­low. Some­times I am even ex­plicit about it. When I save JSON into a file, I name it “foo.json”. But none of that is for­mal­ized in the lan­guage.

And in some way, albeit in a very hacky one, shell is to some ex­tent aware of the types. When I type “ls” and press Tab twice a list of files ap­pears on the screen. When I type “git check­out” press­ing Tab twice re­sults in a list of git branches. So, in a way, shell “knows” what kind of ar­gu­ment is ex­pected.

And the ques­tion that’s bug­ging me is whether the same can be done in a more sys­temic way.

Maybe it’s pos­si­ble to have a shell-like lan­guage with ac­tual type sys­tem. Maybe it could know that file with .json ex­ten­sion is sup­posed to con­tain JSON. Or it could know that “jq” ex­pects JSON as an in­put. Maybe it could know that JSON is a kind of text file and that any pro­gram ac­cept­ing a text file (e.g. grep) can there­fore ac­cept JSON as well. And it could know that “ls -l” re­turns a spe­cific “type”, a re­fine­ment of “text file” and “file with one item per line”, with items like ac­cess rights, own­er­ship, file size and so on.

But how would one do that?

In ad­di­tion to the lan­guage im­ple­ment­ing a type sys­tem it would re­quire some kind of an­no­ta­tion of com­mon UNIX util­ities, adding for­mal speci­fi­ca­tion of their ar­gu­ments and out­puts. (With all pro­grams not pre­sent in the database de­fault­ing to “any num­ber of ar­gu­ments of any type and any out­put”.) Maybe it can be done by sim­ple type-safe wrap­pers on top of ex­ist­ing non-type-safe bi­na­ries.

Pow­erShell does a lot of this, doesn’t it? Pow­erShell aban­dons the con­cept of pro­grams trans­fer­ring data as text, and in­stead has them tran­fer­ring se­ri­al­ized .Net ob­jects (with type an­no­ta­tions) back and forth. It doesn’t ex­tend to the filesys­tem, but it’s en­tirely pos­si­ble to write func­tions that en­force type guaran­tees on their in­put (i.e. re­quiring num­bers, strings, or even more com­pli­cated data types, like JSON).

A good ex­am­ple is search­ing with reg­exps. In Unix, grep re­turns a bunch of strings (namely the lines which match the speci­fied regex). In Pow­erShell, Select-String re­turns match ob­jects, which have fields con­tain­ing the file that matched, the line num­ber that matched, the match­ing line it­self, cap­ture groups, etc. It’s a much richer way of pass­ing data around than de­limited text.

In the early 1990s, there was a com­puter called the Ra­tional 1000, which was pretty much a spe­cial­ised de­vel­op­ment ma­chine for pro­duc­ing code in the Ada pro­gram­ming lan­guage.

The “shell” lan­guage for the sys­tem was… Ada.

It was a weird choice—Ada is com­piled. Ada is a strongly/​strictly typed lan­guage. Ada is cer­tainly not terse, but the IDE helped with a lot of the boiler­plat­ing. It is not what you nor­mally think of as a script­ing lan­guage.

Nonethe­less, I think it was very suc­cess­ful. The users all knew Ada well. The com­mand line (it­self, writ­ten with the help of an IDE) knew all the types of all the pa­ram­e­ters and could help com­plete them (alas—very, very slowly).

I see this as an anec­dote to sup­port your idea that a typed script­ing lan­guage could work.

Make a pow­er­ful enough sys­tem shell with an ex­pres­sive enough pro­gram­ming lan­guage, and you shall be able to unify the con­cepts of GUI and API, herald­ing a new age of user em­pow­er­ment, imo.

This unifi­ca­tion is one of the pro­jects I’d re­ally like to de­vote my­self to, but I’m sort of wait­ing for Rust GUI frame­works to de­velop a bit more (since the shell lan­guage will re­quire a new VM and the shell and the lan­guage server will need to be in­ti­mately con­nected, I think Rust is the right lan­guage). (It may be pos­si­ble to start, at this point, but my fo­cus is on other things right now.)