Thursday, April 26, 2007

NAME

lab 77 - the unexpected markov

NOTES

This is another unexpected use (again) of the markov program
from the `The Practice of Programming', section 3.9 [1].
I wrote an implementation of markov under Limbo, and had fun feeding it
with any texts (books, articles, interviews, novels ...).

But recently I've been also playing with caerwyn's synth [2] which
included in acme-sac, and thought why not feeding markov with music?

Find out the answer by yourselves,
I'll just provide some small hints in the accompanying guide file.

this line can be added to your profile, to have them available when sh starts. After this commands under $home/dis/sh can be executed as any other sh(1) shell command available under /dis.
To make an example now one can checkout inferno-lab running:

FILES

Thursday, April 12, 2007

NAME

lab75 - scope & experiments

DESCRIPTION

Since i did lab 67 i've been trying to improve/fix the t3 port and experiment with it.
So this post has small report of the T3 port status, and some experiments under Inferno; note that they're not dependent of the handheld.

T3 PORT

Some of the thing i've fixed are:

t3 touchscreen:
Perform corrections to make it work as it should, this was important since it has direct impact on using Acme, and the rest of the system.

blank the lcd:
Added code to turn off the lcd while playing music, so battery lasts longer. To do so i added an blank command to the devapm.c written by Alexander Syschev, and wrote a blight script that manages the lcd backlight,to control this from acme.

Since this changes apply to the t3 port, they can be found under lab 67 of the inferno-lab

While i haven't been able to fix the following segfaults, i've been able to obtain dumps and open them with gdb. So i've been able to find where the crashes happen, but still don't understand why they happen, so i'll have to read more.

and if i disassemble emu, i can find that the <addr> corresponds to dopoolalloc(), and the core dump says that the crash happends at /libmath/dtoa.c:/rv_alloc\(i\)/ .

* emu -c1 /dis/sh.dis:
emu starts loading Emuinit /appl/cmd/emuinit.b, it's able to load a few modules on which /dis/sh.dis depends (disdep /dis/sh.dis) but it crashes while loading Filepat, this seems harder since i'm still learning about arm architecture, and i know even less about jit.

EXPERIMENTS

Some of them were begun on lab 67 but i've improved them a bit and it's worth to dedicate them a new post.

scope.b:
Parallel to the work of caerwyn with the DSP toolkit, i thought that it would be interesting to have a flexible oscilloscope, to view signals. So i started with the scope.b of lab 12, and started to add:

look

make it look nicer (add grid, and signal info)

feel

make it more flexible, adding some options and widgets to control the oscilloscope.
Also in this direction: allow resizal of the oscilloscope window.

after some attemps i decided to pack it under a script that could be used both for publish '#A' (server) and to mount '#A' (client), the problem is that if you try, the sound appears delayed and some echo.

Reading stream(1) i noticed that stream ussies writes of bufsize (by default Sys->ATOMICIO: 8192), and to do it has to wait until it has bytes bufsize, so here was the reason for the delay and echo.
Playing with bufsize, i set it by default to 1024 bytes, this works on a local net,
probably it'll need adjustment for other situations.

player:
i started this as a music player for the handheld, but i've ended using it under the PC, it's simply a script that lists the music files at cwd and plays them with a music player available in the host os: mplayer.
A nice thing is that one can export a dir containing a collection of music files with:

styxlisten -A 'tcp!*!styx' export /n/local/music

mount them from another machine under /n/remote
and play them running:

mount -A 'tcp!$server!styx' /n/remote
cd /n/remote && player

kbd2skini.b:
This is just an idea in the direction of the DSP toolkit, currently the DSP toolkit
is able to generate music, from standard music formats mainly contained in files (midi,skini).
I would like to play with the input devices available under inferno to use them as instruments. So i started to gather for similar things and found pc2midi (and some useful tutorials).

I've choosen the kdb as starting point, but it has it's limitations as it's discussed on the previous reference, since you don't have a pressure indication, instead the state of each key is either pressed / or not pressed, moreover there's no way to notice that there're multiple keys pressed at a time.
We'll see what can be done with this, but could this be more than a joke?

FILES

Saturday, April 07, 2007

NAME

NOTES

I read the Effective Java book recently.
Every language needs a book that describes how to use it effectively.
I wish there was a book for Effective Inferno.

Although common techniques may work in Inferno
I know for sure there are some uncommon ones that
may work better.

I'll try and describe at least one
recipe that could be a chapter in that book.

To people who've asked me what distinguishes Inferno,
I've answered that it's the concurrent language Limbo,
or that it's a portable OS that runs on a VM,
or that it uses the Styx protocol to create a distributed system,
or that it's about software tools that work together with the Inferno shell.

These are the ingredients; on their own they are not unique to inferno.
Limbo looks very like C and to a newbie it's
not obvious what is special about it.
There a lots of little niceties that
make Limbo pleasant to use, e.g., array slicing and tuples,
but these are details; they do not
greatly impact an application architecture.

A lot of the Unix-like commands under /appl/cmd/
are straight ports from Plan 9. This is old school.
And the fact that they're implemented in Limbo does not
greatly differentiate them from their C counterparts.
The special flavor of Limbo does not come out when used this way.

The OS interface also looks much like Unix or Plan 9, except for the VM piece,
so what is special about that? The advantages of platform
independent come out when doing distributed applications. So there
must be techniques that need to be learned to do that effectively.

In Inferno it's a new world where each ingredient
might be familiar to you in another context but we can cook things
a little differently here.

Extension Points

For any system we need to know how we might extend it and how we
combine the pieces that are already there. Inferno provides
a number of interfaces at different levels of granularity where the
programmer can extend the system. Lets quickly review the most common
ones, available in most systems.

1. We can write libraries and extend the systems with new functions; such
things as parsing file formats or protocols.
We can't live without those libraries. Most of the ones
in /appl/lib are of this form.

2. We can create new commands and add to our software tool set.
Commands can be developed that do one thing well, and grow the
system organically.

3. We can use the shell to combine commands
using pipelines or to customize command interfaces or extend
the system further with commands implemented in the shell.

These are the contributions from our Unix legacy. And they are great!
Plan 9 followed and gave us everything-is-a-file, the 9p protocol,
and private namespaces. Inferno, being a direct descendant of Plan 9,
inherited all that.

4. We can build filesystems to provide new services.
We can also architect a distributed system by
basing applications around the Styx protocol.

We don't have to use Limbo to extend
inferno in this way, we can use any language, and there are already several
implementations of the styx library. The styx protocol is itself an
incredibly powerful and flexible way of extending the system. It permits
re-use of existing tools that operate on files.

5. We can extend the system as clients to file services.
Two big examples from Inferno
and Plan 9 are Acme and the Plumber. Because of the loose coupling
between client and server we can extend the system while it is running.
We can write new Acme plugins while we are working within Acme.

This is a great pattern for developing services that you want extended.
Don't just be a client of the system extention points but create your
own extension points.

You may think this is where I'd stop. This would be enough right?
What a great system. Plan 9 is a lovely system for all the above reasons.
But we haven't got unique to inferno yet. And there is something more lovely.
Bear with me, this will take a few paragraphs.

6. Limbo supports dynamically loadable modules.
We can define an extension
point as a module interface that the client is expected to implement.
Then we can load the client, one of possibly many implementations,
as needed.

This is distinct from a module interface used by libraries
where in practice there is really the one
and final implementation. [1], [2].

What I mean here is that there a many clients that might do distinctly different
things but are able to use a shared interface with other clients.

The simplest example is the Command interface with the shell controlling
loading of modules.

This is the extension point. You implement a client that supports this interface
and then the shell will call it. Implicit in the interface is the fact that you run
as a separate process and you have file descriptors for stdin, stdout, and stderr.

Of course, we all know this interface because it's similar to ones used
by shells on most operating systems.

Now you say, I've already covered this in point 2.
But this is different.
Inferno allows us to define new interfaces, that can be specific to our problem.

For example, the Inferno shell defines a more complicated interface
for shell loadable modules
that extend the shell and use shells syntax to for structured arguments.

But here's the kicker. Inferno's shell implements the Lispish pattern of using
a standard syntax to represent data and code. Instead of parens it uses braces
but other than that we have lists of nested lists.

{f x {g y z } }

7. We can extend the system by re-using shell syntax to structure
the input to our own applications

The shell parses
a block, does syntax checking but doesn't eval it.
It leaves it up to the
command to evaluate, either builtin or external.
So we can implement the shell pattern,
but not use the shell builtin infrastructure.
We define our own evaluator and
pass it a shell block.
We define, in a sense, our own shells specific to a problem domain.

The other big piece to this is that the module interfaces can include
typed channels and that each run in a separate process so
that the modules can communicate to each other, working
together and forming
the equivalent of pipelines, but with types.

I'll give a few examples to really push this point. This is not a trivial feature
and it's easy to overlook.

The prime example in inferno is the fs(1) command.

Fs uses shell syntax but doesn't use shell to evaluate it. Fs is a tree walker
that allows arbitrary mixing of components to operate on a tree.
It's a remarkable command because it combines sh syntax reuse, with
the module extension point for clients that use channels to communicate
between processes. Its a lovely thing.
The extension point looks like this.

Here's an example of how I use fs to build the acme-sac distribution. I filter out .svn files, filter out .sbl and .dis files below /appl and object files below /sys, then copy the whole tree using a proto file to define the components to copy.

It's very lispish in that we've built a domain language, in this case for tree walking,
which we can add to, a command at a time, re-using existing commands
in possibly novel ways, and using shell syntax
to represent the whole expression.

By combining loadable modules, sh syntax, a uniform interface, channels, processes
and files, we have a unique programming environment. This is a powerful
pattern that should be repeated.

This kind of programming is now quite unlike any other. It establishes
a pattern for structuring applications. It leverages some of the quality
ingredients inside inferno.

Another example is the sound synthesizer I've been working on.
I'm consciously imitating fs here.
I use a simple module extension point with processes communicating
on channels, a sh syntax that combines the modules.

We're still not done. We haven't used all the ingredients. Lets
throw in Styx.

Lets combine all the above, plus the fact
that dis is completely platform independent so we can compile
once and run anywhere that inferno emulator runs.

The example that would illustrate this is VN's grid. But I'm guessing
since I haven't used it. You can also see all these ingredients combined
in sh-alphabet's grid. Sh-alphabet is quintessential Inferno.

From these ideas I'm trying to implement a mapreduce framework.
I will use sh syntax to form the expression
dynamically, use module extensions to write new map and reduce functions,
use Styx to create the mapreduce infrastructure, use dis to distribute
code and have channels and files for communicating between processes.

The mapreduce command is itself a file system that is exported to remote hosts
so that clients read and write to it, reading instructions on what to do next,
writing back status, so the master can keep track of what's going on. Command
usage might be as follows,

I'll stop here, even though I still haven't played the polymorphism card.
Sh-alphabet does, but it's a complicated example to swallow.

I recommend programmers start by using fs(1). Then look at
how your own programs can be re-worked to factor out modules
using a module extension point that uses channels, and then use sh syntax
for combining those modules.

The next step may then be to distribute those modules by building into it a styx service.

FOOTNOTES

[1] Actually, there is a slight variation in the library idea
in that a library can have multiple implementations, a kind of polymorphism.
The Imagefile interface is of this form. This is a slightly more sophisticated
library where we load a particular library depending on the kind of operation.
But a user of the system is not really expected to extend the system at these points,
even though they might. (Other examples are Filter and Encoding)

[2] Another variation on the command polymorphism is that we can bind alternative
implementations over our namespace. But in practice this seems to be done rarely.
Binding /acme/dis/cd.dis over /dis/cd.dis is one example. Maybe this could be
exploited further.