For the OPAM users among you, I've added a 4.01.0 compiler switch
which will track the RC releases, and eventually become the official
release when it's out.
$ opam update
$ opam switch 4.01.0
$ eval `opam config env`
# ???
# profit
I've been running a full regression test of the entire OPAM database
on a variety of architectures and triaging the results. If anyone
else is interested in helping submit patches to fix issues, then
please see:
https://github.com/OCamlPro/opam-repository/issues/1029
Note that the above link is a bit raw, since it also includes failures
from bugs in 4.00.1 packages too. We've just been fixing everything
we can spot in preparation for testing the forthcoming OPAM 1.1 beta
on as stable a package database as possible. I'm preparing a blog
post describing the most common failures we've seen to help package
authors avoid them in the future (warnings-as-errors, I'm looking at
you!)

Literate (sort-of) programming in OCaml

I am currently writing a big, mostly textual document in format blah
(blah being of a course a meta-variable). Are there any tools that
would allow me to interleave OCaml code with the contents of my
document?
More on my use-case. Creating this document requires me to perform
various subtasks, such as:
- write a code snippet in the document,
- put the code snippet in a file,
- call an external program on the file,
- paste the output into the document.
Naturally, I wish to automate this. I could roll my own set of
commands, and parse them with OCaml, but I would be re-creating a
scripting language, and it seems to me that interleaving OCaml code
within my document would be better. Here's what I have in mind:
blah blah blah blah blah blah blah blah blah blah blah blah blah blah
blah blah
blah blah blah blah blah blah blah blah blah blah blah blah blah blah
blah blah
blah ... we thus write the following code ... :
{%
let code = "<sample code>" in
output code
%}
blah blah blah ... after running the command blah ... the output is as
follows ... blah
{%
let f = write_into_temp_file code in
let s = run_and_read "myprogram" [f] in
output s
%}
blah blah blah blah blah blah blah blah blah blah blah blah blah blah
blah blah ...
Are you aware of any tool that would allow me to achieve this?

Alain Frisch suggested:

I'm sure you can use sed to (I use '' for meta quotes):
- Add ''print_string"'' in front of the file and ''";;'' at the end.
- Replace ''{%'' with ''";;'' and ''%}'' with ''";; print_string"''
If you apply this transformation and call the toplevel on the result,
you should get what you want, no?

Current workflow I use involves:
- mkfile: a rule file for mk (Plan9's take on make, it has simple
escaping and a readable man page) to drive the whole thing,
- bar: a compiler for the language bar that includes a Latex pretty
printing mode
- inc/: a directory where I place all my code samples from language bar
- main.pdc: a pandoc file holding the source.
Extracts from the mkfile:
~~~~~~~
TARGET=main.pdf
SOURCE=${TARGET:%.pdf=%.pdc}
INCLUDES=`{cat $SOURCE | grep '\\input{.*\.tex}' | sed
's/^.*\\input{([^\$\\}]*\.tex)}.*$/\1/'}
$TARGET: $SOURCE $INCLUDES
pandoc -o $TARGET $SOURCE
%.pp.tex: %.bar bar.byte
OCAMLRUNPARAM=$OCAMLRUNOPT ./bar.byte -latex $stem.bar > $target
%.pp.annot.tex: %.bar bar.byte
OCAMLRUNPARAM=$OCAMLRUNOPT ./bar.byte -latex -annot $stem.bar > $target
~~~~~~~
Extracts from bar.mli:
~~~~~~~
module type PRINTER = sig
val t: annotations:bool -> t -> string
end
module LatexPP : PRINTER
~~~~~~~
Extracts from the main.pdc
~~~~~~~
The annotation process, blahblah. Thus the program $\input{inc/foo.pp.tex}$
is annotated in the following way: $\input{inc/foo.pp.annot.tex}$
~~~~~~~
Known bugs and shortcomings:
- sed works line-wise so the mkfile INCLUDES variable is not set
properly when there are several \input on the same line.
- the passing of argument (using two rules with and two file
extensions) is hackish and would not scale.
Less painful than copy-pasting things and easier to keep everything in
sync. It's also nice to be able to do things such as:
\begin{align*}
<some math>\input{inc/function-application.pp.tex}<more math>\\
<more math>\input{inc/function-declaration.pp.tex}<more math>\\
<more math>\input{inc/if-then-else.pp.tex}<more math>\\
<etc.>
\end{align*}

Hmhhh, long, long ago I explored some of the good-old / classical
literate programming tools (well, to be detailed, I did not tried Web,
so it was not THE classical one, but later developments, influenced by
Knuth's Web.)
Some of these tools were limited to certain languages, but as far as I remember
at least one was able to work with any (???) language.
Or at least some tools were very flexible to work with many languages.
I have forgotten most of the names of the tools I used, possibly I tried
noweb and some others.
All had their advantages and also their limitations.
There was one (or two?) that looked so promising that I had planned to try it
(them) too, but I stopped with Literate Programming (LP) before I had the
motivation to try it/them.
(Trying so many tools at some time has exhausted and demotivated me.
Those I tried were not powerful/flexible enough, nevertheless needed their
time to look at them. Tools should help in the work and help saving time,
not eat the time...)
As far as I remember, and I'm not sure, FWEB and/or FunnelWeb was/were the
remaining tools that I had selected as the most prommising ones (but did not
tried them, as mentioned above, so I can't really vote for them).
(In/with R you can use Sweave, which allows to embed code and results from code
like tables, and even to embedd created graphics into the document!)
An overview with links to some of the LP-tools can be found here:
http://www.literateprogramming.com/tools.html
I remember that there were a lot more free tools, than mentioned there.
I remember one really nice tool only for C-dialects, which did not have
big functionality, but allowed to pretty print C-Code with real TeX-Comments
inserted. And the syntax was very simple.
But it was not allowing insertion of the results of the program,
at least not directly. But via TeX/LaTeX-include/input--commands I think it was possible
indirectly.
But you look for more functionality, and want to pass back the results into
your document. Something like Sweave.
Possibly FWEB or FunnelWEB are the tools, you should explore first.
(But be prepared to learn their syntax....)
Let me know, which tool you selected, and how your experience was with it.
Ciao,
Oliver
P.S.: In a documenting-/reverse-engineering project I programmed my own
literate programming tool(s), so that I could insert Text and
graphviz-commands into the code-comments. The Perl-Tools did extract the
text and the graphviz code, invoked graphviz and created a LaTeX-document
that contained the Code, the documenting text and the Graphviz-graphics
(e.g. for Flowcharts), and created a nice pdf-document, including the
nice references, to open certain parts of the doc by clicking on the
entries in the table of contents. :-)
By organizing the document via includes/inputs, the document structure could be
organized the way I wanted. (The laTeX-code was also created by the
Perl-script(s).
P.P.S.: When using troff it's possible to write C code and it's documentation
in troff (including PIC-macros :-)) in a way, that it can be C-compiled
directly, without tangling-process, and the document can be set with
troff without weaving-process. So, it's possible to have one C-file and
pass it into C-compiler and troff without changes. :-) Very funny, to
make this. But changing the order of sections is not possible. Real
LP-tools IMHO should offer reordering of sections, but most of them
don't.

Alan Schmitt suggested:

I'm using org-mode for this (http://orgmode.org/). The support for caml
uses a session, so when you evaluate / export code the previous code is
taken into account. I've used it for courses slides (exported to beamer)
and for caml exams (where the type of the expected answers is computed
from the correction, which is of course not exported).

Jonathan Protzenko replied and Alain Frisch added:

> Thanks for the numerous replies. There are a lot of good solutions in
> there, and the ones that looked closest to what I had in mind seem to be
> MPP and the rwo-run-toplevel script. Some questions about these two tools:
> - MPP seems to just have the built-in language. I couldn't find any
> example in the examples/ directory that used another language, such as
> OCaml. I guess that's because it's still in development?
> - rwo-run-toplevel seems closer to what I have in mind; if I could send
> all the ocaml parts to a top-level session, and put in my document what
> gets sent to stdout (not the toplevel reply, that is, not the "val f:
> int -> int" parts), that would be pretty much all I need.
> rwo-run-toplevel seems to perform a lot of work, though: what are all
> these rewrite rules?
>
> Other solutions seemed interesting. The Perl script scared me a little
> bit; being no Perl hacker, I would have a hard time using it. The
> Markdown library does not seem to be an exact fit (I want the output of
> arbitrary OCaml commands to be put in the middle of my markdown.) Stog
> seems to be oriented towards reproducing an interactive toploop session,
> while I'm more interested in executing arbitrary commands. Alain's sed
> trick seems to be exactly what I need, I'm just a bit afraid if I ever
> need to change something, it'll be break. But in essence, yes, that's
> just all I need. Other proposals seemed a little bit too far away from
> what I had in mind.
Be careful with OCaml lexical convention though. If you use e.g. " or \
in your text, you'll need also to escape them with sed. Alternatively,
you could use the new syntax for string literals ({foo|....|foo}) now
available in trunk.

Philippe Wang also replied:

MPP currently has 2 things: the built-in language and its commands,
and the ability to bring OCaml as a pre-processor language.
With mpp -its (-its is an option to be less strict on trailing spaces
in commands) :
((x cmd ocamlopt foo.ml x))
will execute "ocamlopt foo.ml" in a shell and input its output.
Note that "x" is the name of the block. It could be FOO or 23NUI'è!çé
instead, so that one can write ((x cmd echo $((42+42)) x)) for
instance without being bothered by escaping stuff.
-> This is an example of the builtin commands.
To have the list of built-in commands: mpp -b
One may want to nest mpp commands. In this case, another pair of
tokens is used: {{ and }}.
so ((x cmd ocamlopt foo.ml x)) could be written {{x cmd ocamlopt
foo.ml x}} instead. In this precise case, there is absolutely no
difference. However, one can also write {{x cmd ocamlopt {{y get
filename y}} x}} or {{x cmd ocamlopt ((y get filename y)) x}}. But if
one write ((x ocamlopt ((y get filename y)) x)) then a shell will try
to run "ocamlopt ((y get filename y))", which is probably not what's
intended...
All tokens of the language provided by mpp are customisable on the
command line and live in mpp as well. mpp -so 42 will use "42" instead
of "((" to open blocks.
(( so TOTO)) will use TOTO to open blocks after this one, meaning that
to set the open token back to what it was, one could write TOTO so
(())
Well, it's up to the user to imagine how powerful it is.
Then, there is the ability to bring ocaml as a pre-processor language.
This is actually the main motivation for mpp.
Sometimes, you write something using a specific language and you just
wish you could easily use OCaml (or another language...) to preprocess
your file.
With mpp, the default tokens are {< and >} to embed ocaml.
This means that in any file, one can write ocaml and have its output.
Small example in markdown:
If you write this in the file e.md
```
# Number of the day
{< let foo = 42 >}
The number of the day is {< let () = Printf.printf "%d" foo >}.
```
And you run mpp -l ocaml over e.md (note the -l option), then you obtain
```
let _ = print_string "# Number of the day\n\n"
let foo = 42 let _ = print_string "\nThe number of the day is \n"
let () = Printf.printf "%d" foo let _ = print_string ".\n\n"
```
which is an ocaml programme that you can run to have your final document:
```
# Number of the day
The number of the day is
42.
```
You can see that the environment is shared. If you want different
environments, it's possible to run mpp several times with different
opening and closing tokens.
Of course, you may use ocaml as a preprocessor language in an ocaml
programme (instead of a markdown document), or any textual file
actually.
For the moment, ocaml is the only option supported by "mpp -l" but
I'll provide more language soon. (To provide a language X, one has to
write a "description" of how to convert a text file into an X
programme.)
I've not written much documentation for MPP yet because I've been
mainly working on OMD, a Markdown->HTML converted and Markdown parser
library in OCaml. Now I believe OMD is nearly done, so I'm getting
back to MPP soon...

Pippo: a pretty interesting pre-processor that uses OCaml

Following yesterday's discussion, and using that as a pretext for
writing my own tool, here's yet another pre-processor-sort-of.
https://github.com/protz/pippo
It requires OCaml 4.01.0rc1. The documentation for it is, of course,
pre-processed using Pippo. It is distributed with a Makefile that
allows you to recompile the documentation and thus play with the tool.
It works as described in yesterday's thread: anything found between {%
and %} is fed to an OCaml interactive session. Therefore, the output of
the OCaml commands is interleaved with the non-OCaml parts of the file.
Basically, the tool allows you to script your documents easily.
The tool also contains a special "inject_value" facility, that allows
you to make any OCaml value (constant, function) available in the
top-level session. For instance, running:
inject_value
"__version"
"unit -> unit"
(fun () ->
print_endline "This is pippo v0.1");
will allow you to write, in the to-be-pre-processed document :
{%
__version ();
%}
For the record, the question (how do I use Toploop.setvalue) was asked
a few years ago but left unanswered
http://caml.inria.fr/pub/ml-archives/caml-list/2009/03/7e9085b8601142024108e254df9dfb1e.en.html.
The OCaml maintainers should probably not look at the implementation,
by the way.
Cheers,
~ jonathan
PS: My warmest thanks go to Thomas Braibant for helping me find a name
with a pun (in French)