Our team's products are built with Flow, a functional language in the ML family. Flow lets us write code once and deliver it to students on multiple platforms and device types. Other languages in our development ecosystem include especially JavaScript, but also C++, SWF (Flash), and Haxe.

If you're interested in functional languages like Scala, Swift, Erlang, Clojure, F#, Lisp, Haskell, and OCaml, then you'll enjoy learning Flow. We don't require that you have previous experience with functional programming, only enthusiasm for learning it. But if you have do some experience with functional languages, so much the better! (On-the-job experience is best, but coursework, personal projects, and open-source contributions count too.)

We require only that you:

Have a solid grasp of CS fundamentals (languages, algorithms, and data structures)

Be comfortable moving between multiple programming languages

Be comfortable with modern software practices: version control (Git), test-driven development, continuous integration, Agile

The solution should be easy enough to obtain but its time complexity is O(n^2) which is worse than the required O(nlogn).

Introducing Divide and Conquer

The algorithm design technique divide and conquer was mentioned in Recursion Reloaded. I believe it is a good time to properly introduce it now as it provides a elegant approach towards a better solution for pearl 2.

What is it, literally

Suppose we want to replace the dragon lady and become the king of the land below ([2]).

We are very lucky that we have got a strong army and now the only question is how to overcome the realm.

One "good" plan is no plan. We believe in our troops so much that we can just let all of them enter the land and pwn everywhere.

Maybe our army is very good in terms of both quality and quantity and eventually this plan will lead us to win. However, is it really a good plan? Some soldiers may march to places that have already been overcome; Some soldiers may leave too soon for more wins after one winning and have to come back due to local rebel... the whole process won't be efficient and it cost too much gold on food for men and horses.

Fortunately, we have a better plan.

We divide the land into smaller regions and further smaller ones inside until unnecessary. And for each small region, we put ideal amount of soldiers there for battles. After soldiers finish their assigned region, they don't need to move and just make sure the region stay with us. This is more oganised and more efficient in terms of both gold and time. After all, if we conquer all the tiny regions, who would say we were not the king?

What is it in algorithm design, with accumulation

divide and conquer in algorithm design is not a universal solution provider, but a problem attacking strategy or paradigm. Moreover, although this classic term has been lasting for quite a long time, personally I would like to add one more action - accumulate - to make it appear more complete. Let's check the 3 actions one by one to see how we can apply the techque.

Divide

Conceptually this action is simple and we know we need to divide a big problem into smaller ones. But how to is non-trivial and really context-sensitive. Generally we need to ask ourselves 2 questions first:

What are the sizes of the smaller sub-problems?

Normally we intend to halve the problem because it can lead us to have a O(logn) in our final time complexity.

But it is not a must. For example 3-way quicksort divides the problem set into 3 smallers ones. 3-way partition can let quicksort have O( $ \log_3{n} $ ). However, do not forget the number of comparison also increases as we need to check equality during partition.

Moreover, sometimes we may have to just split the problem into a sub-problem with size 1 and another sub-problem with the rest, like what we did for the sum function. This kind of divide won't give us any performance boost and it turns out to be a normal recursion design.

Do we directly split the problem, or we somehow reshape the problem and then do the splitting?

In mergesort, we simply split the problem into 2; while in quicksort, we use partition to rearrange the list and then obtain the 2 sub-problems.

The point of this question is to bear in mind that we do not have shortcuts. We can have a very simple splitting, but later on we need to face probably more complicated accumulate. Like mergesort relies on merge. Or, we can do our important work during divide phase and have a straight accumulate (quicksort just needs to concatenate the solutions of the two sub-problems with the pivot in the middle).

Conquer

This action implies two things:

Recursion. We divided the problem, then we need to conquer. How to conquer? We need to apply divide and conquer and accumulate again until we are not able to divide any more.

Edge cases. This means if we cannot divide further, then it is time to really give a solution. For example, let's say our target is a list. If we reach an empty list or an one-element list, then what shall we do? Normally, if this happens, we do not need to accumulate and just return the answer based on the edge cases.

I believe the conquer part in the original divide and conquer term also implies the accumulate. I seperate accumulate as explained next.

Accumulate

After we conquer every little area of the land, we should now somehow combine all our accomplishments and really build a kingdom out of them. This is the step of accumulate.

A key way to figure out how to accumulate is to start from small. In mergesort, if each of the 2 sub-problems just has one element, then the according answer is a list having that element and we have finished the conquer step. Now we have two lists each of which has one element, how can we accumulate them to have a single sorted list? Simple, smaller element goes first into our resulting list. What if we have two sorted list each of which has two elements? The same, smaller element goes first again.

If we decide to divide the problem in a fairly simple way, then accumulate is normally non-trivial and also dominates the time complexity. Figuring out a cost-efficient approach of accumulate is very important.

Summary

Again, divide and conquer and accumulate is just a framework for solving an algorithm problem. All the concrete solutions are problem context based and can spread into all 3 steps.

In addition, a fundamental hint to using this techqniue is that if we are given a problem, and we know the future solution is not anyhow related to the problem size, then we should try divide and conquer and accumulate

Solve pearl 2

Although pearl 2 asks us to get the max number of surpassers, we can

Get the number of surpassers for every element (we anyway need to)

Then do a linear scan for the max one.

The second step is O(n). If we can achieve the first step in O(nlogn), the overall time complexity stays as O(nlogn).

So our current goal is to use divide and conquer and accumulate to get all numbers of surpassers.

Divide the problem of pearl 2

We have such as list and we want to get a new list that have the same elements and each element is associated with the number of its surpassers. Now we want to divide the original list (problem set).

Can we directly halve the list?

As pearl 2 stated, an element only cares about all elements that are behind it. So if we split the list in the middle, we know the numbers of surpassers for all elements in sub-problem 2 do not need any special operations and the answers can directly be part of the future resulting list.

For the elements inside sub-problem 1, the answers are not fully accomplished yet as they will be affected by the elemnts in sub-problem 2. But hey, how we obtain full answers for sub-problem 1 with the help of the solutions of sub-problem 2 should be the job of accumulate, right? For now, I believe halving the problem is a good choice for divide as at least we already solve half of the problem directly.

Conquer

We of course will use recursion. For sub-problems, we further halve them into smaller sub-problems until we are not able to, which means we reach the edge cases.

There are possibly two edge cases we need to conquer:

Empty list

A list with only one element.

For empty list, we just need to return an empty list as there is no element for us to count the number of surpassers. For an one-element list, we also return an one-element resulting list where the only element has 0 surpassers.

Accumulate

Now we finished dividing and conquering like below and it is time to accumulate (take sub-problem 1 only for illustration).

It is easy to combine solutions of sp 111 and sp 112: just compare 8 from sp 111 with 1 from sp112, update 8's number of surpassers if necessary and we can leave sp 112 alone as we talked about during divide. The same way can be applied on sp 121 and sp 122. Then we get:

Now both sp 11 and sp 12 have more than one element. In order to get the solution for sp 1, sp 12 can stay put. How about sp 11? An obvious approach is just let every element in sp 11 to compare every element in sp 12, and update their numbers of surpassers accordingly. This can be a candidate for accumulate action, however, it is O(n^2). We need to accumulate better.

We said in the very beginning of this post during our trivial solution that the original order of the list matters. However, is it still sensitive after we get the solution (for a sub-problem)?

As we can see once the answer of sp 11 is obtained, the order between 8 and 1 doesn't matter as they don't rely on each for their number of surpassers any more.

If we can obtain the solution in a sorted manner, it will help us a lot. For example, assume the resulting lists for sp 11 and sp 12 are sorted like this:

Then we can avoid comparing every pair of elements by using merge like this:

We can see that 8 in the left hand side list doesn't have to compare to -10 any more. However, this example has not shown the full picture yet. If keep tracking the length of resulting list on the right hand side, we can save more comparisons. Let's assume both sp 1 and sp 2 have been solved as sorted list with lengths attached.

We begin our merge.

Have you noticed the fascinating part? Because -10 < -2, without further going down along the resulting list on the right hand side, we can directly update the number of surpassers of -10 and get it out. Why? Because -2 is the smallest element on the right, and if it is bigger than -10, then the rest of the elements on the right must all be bigger than -10, right? Through only one comparison (instead of 4), we get the number of surpassers.

Thus, as long as the solutions of all sub-problems are sorted list with the length associated, we can accumulate like this:

Compare the heads hd1 and hd2, from two sorted resulting lists l1 and l2, respectively

If hd1 >= hd2, then hd2 gets out; go to 1 with updated length for l2

if hd1 < hd2, then hd1 gets out, and its ns gets updated by adding the length of l2 to the existing value; go to 1 with updated length for l1

The full process of accumulating sp 1 and sp 2 is illustrated as follows:

Two things might need to be clarified:

Although we assumed the resulting lists of sub-problems to be sorted, they will naturally become sorted anyway because we are doing the smaller goes first merging.

We need to attach the lengths to each resulting list on the right and keep updating them because scanning the length of a list takes O(n).

Obviously this way of accumulation can give us O(n). Because at most we can divide O(logn) times, our divide and conquer and accumulate solution will be O(nlogn).

Code

At first, we divide. Note that this version of divide is actually a kind of splitting from middle, as the original order of the elements before we get any solution is important.

(* we have a parameter n to indicate the length of l.
it will be passed by the caller and
in this way, we do not need to scan l for its length every time.
it will return left, right and the length of the right.
*)
let divide l n =
let m = n / 2 in
let rec aux left i = function
| [] -> List.rev left, [], 0
| right when i >= m -> List.rev left, right, n-i
| hd::tl -> aux (hd::left) (i+1) tl
in
aux [] 0 l

Now accumulate. We put it before writing conquer because conquer would call it thus it must be defined before conquer.

Are we done? No! we should find the max number of surpassers out of them:

(* we should always consider the situation where no possible answer could be given by using **option**, although it is a bit troublesome *)
let max_num_surpassers = function
| [] -> None
| l ->
let nss = numbers_of_surpassers l in
Some (List.fold_left (fun max_ns (_, ns) -> max max_ns ns) 0 nss)

[1] Unless I can see an optimal solution instantly, I always intend to think of the most straightforward one even though it sometimes sounds stupid. I believe this is not a bad habit. Afterall, many good solutions come out from brute-force ones. As long as we anyway have a solution, we can work further based on it and see whether we can make it better.

Last week, we
published
an alpha version of a new OCaml documentation generator,
codoc 0.2.0.
In the 2014 OCaml workshop presentation (abstract, slides, video),
we mentioned the 'module wall' for documentation and this attempts to fix it.
To try it out, simply follow the directions in the README on that repository,
or browse some samples of the current,
default output of the tool. Please do bear in mind codoc and its constituent
libraries are still under heavy development and are not fe…

Last week, we
published
an alpha version of a new OCaml documentation generator,
codoc 0.2.0.
In the 2014 OCaml workshop presentation (abstract, slides, video),
we mentioned the 'module wall' for documentation and this attempts to fix it.
To try it out, simply follow the directions in the README on that repository,
or browse some samples of the current,
default output of the tool. Please do bear in mind codoc and its constituent
libraries are still under heavy development and are not feature complete.

codoc's aim is to provide a widely useful set of tools for generating OCaml
documentation. In particular, we are striving to:

Cover all of OCaml's language features

Provide accurate name resolution and linking

Support cross-linking between different packages

Expose interfaces to the components we've used to build codoc

Provide a magic-free command-line interface to the tool itself

Reduce external dependencies and default integration with other tools

We haven't yet achieved all of these at all levels of our tool stack but are
getting close. codoc 0.2.0 is usable today (if a little rough in some areas
like default CSS). This post outlines the architecture of the new system to
make it easier to understand the design decisions that went into it.

The five stages of documentation

There are five stages in generating documentation from OCaml source
code. Here we describe how each was handled in the past (using
OCamldoc), the present (using our current prototype), and the future
(using the final version of the tools we are developing).

Associating comments with definitions

The first stage is to associate the various documentation comments in
an .ml or .mli file with the definitions that they correspond to.

Past

Associating comments with definitions is handled by the OCamldoc
tool, which does this in two steps. First it parses the file using the regular
OCaml parser or camlp4, just as in
normal compilation. It uses the syntax tree from the first step and then
re-parses the file looking for comments. This second parse is guided by the
location information in the syntax tree; for example if there is a definition
which ends on line 5 then OCamldoc will look for comments to attach to that
definition starting at line 6.

The rules used for attaching comments are quite intricate and whitespace
dependent. This makes it difficult to parse the file and attach comments
using a single parser. In particular, it would be difficult to do so in
a way that doesn't cause a lot of problems for camlp4 extensions. This
is why OCamldoc does the process in two steps.

A disadvantage of this two-step approach is that it assumes that the
input to any preprocessor is something which could reasonably be read by
the compiler/tool creating documentation, which may not always be the
case.

Present

Our current prototype associates comments with definitions within the
compiler itself. This relies on a patch to the OCaml compiler
(pull request #51 on GitHub).
Comment association is activated by the -doc command-line flag. It
uses (a rewritten version of) the same two-step algorithm currently
used by OCamldoc. The comments are then attached to the appropriate node
in the syntax tree as an attribute. These attributes are passed through
the type-checker and appear in .cmt/.cmti files, where they can be
read by other tools.

Future

We intend to move away from the two-step approach taken by OCamldoc. To
do this we will need to simplify the rules for associating comments with
definitions. One suggestion was to use the same rules as attributes,
however that seems to be overly restrictive. So the approach we hope to
take is to keep quite close to what OCamldoc currently supports, but
disallow some of the more ambiguous cases. For example,

val x : int
(** Is this for x or y? *)
val y : float

may well not be supported in our final version.
We will take care to understand the impact of such design decisions and we
hope to arrive at a robust solution for the future.
By avoiding the two-step
approach, it should be safe to always turn on comment association rather
than requiring a -doc command-line flag.

Parsing the contents of comments

Once you have associated documentation comments with definitions, you must
parse the contents of these comments.

Past

OCamldoc parses the contents of comments.

Present

In our current prototype, the contents of comments are parsed in the
compiler, so that the documentation attributes available in
.cmt/.cmti files contain a structured representation of the
documentation.

Future

We intend to separate parsing the contents of documentation comments
from the compiler. This means that the documentation will be stored as
strings within the .cmt/.cmti files and parsed by external
tools. This will allow the documentation language (and its parser) to
evolve faster than the distribution cycle of the compiler.

Representing compilation units with types and documentation

The typed syntax tree stored in .cmt/.cmti files is not a convenient
representation for generating documentation from, so the next stage is
to convert the syntax tree and comments into some suitable intermediate
form. In particular, this allows .cmt files and .cmti files to be
treated uniformly.

Past

OCamldoc generates an intermediate form from a syntax tree, a typed
syntax tree, and the comments that it found and parsed in the earlier
stages. The need for both an untyped and typed syntax tree is a
historical artefact that is no longer necessary.

Present

Our current prototype creates an intermediate form in the
doc-ock library. This form can be
currently be created from .cmti files or .cmi files. .cmi files do
not contain enough information for complete documentation, but you can
use them to produce partial documentation if the .cmti files are not
available to you.

Future

Resolving references

Once you have a representation for documentation, you need to resolve
all the paths and references so that links can point to the correct
locations. For example,

(* This type is used by {!Foo} *)
type t = Bar.t

The path Bar.t and the reference Foo must be resolved so that the
documentation can include links to the corresponding definitions.

If you are generating documentation for a large collection of packages, there
may be more than one module called Foo. So it is important to be able
to work out which one of these Foos the reference is referring to.

Unlike most languages, resolving paths can be very difficult in
OCaml due to the powerful module system. For example, consider the
following code:

Here it looks like, Dep2(Dep1).B.c would be defined by a type
definition c within the submodule B of the functor Dep2. However,
Dep2.B's type is actually dependent on the type of Dep2's Arg
parameter, so the actual definition is the class definition within the
module type S of the Dep1 module.

Past

OCamldoc does resolution using a very simple string based lookup. This
is not designed to handle collections of projects, where module names
are not unique. It is also not sophisticated enough to handle advanced
uses of OCaml's module system (e.g. it fails to resolve the path
Dep2(Dep1).B.c in the above example).

Present

In our current prototype, path and reference resolution are performed by
the doc-ock library. The implementation
amounts to a reimplementation of OCaml's module system that tracks
additional information required to produce accurate paths and references
(it is also lazy to improve performance). The system uses the digests
provided by .cmti/.cmi files to resolve references to other modules,
rather than just relying on the module's name.

Future

There are still some paths handled incorrectly by doc-ock-lib, which
will be fixed, but mostly the final version will be the same as the
current prototype.

Producing output

Finally, you are ready to produce some output from the tools.

Past

OCamldoc supports a variety of output formats, including HTML and
LaTeX. It also includes support for plugins called "custom generators"
which allow users to add support for additional formats.

Present

codoc only supports HTML and XML output at present, although extra output
formats such as JSON should be very easy to add once the interfaces settle
down. codoc defines a documentation index XML format for tracking package
hierarchies, documentation issues, and hierarchically localized configuration.

codoc also defines a scriptable command-line interface giving users access
to its internal documentation phases: extraction, linking, and rendering. The
latest instructions on how to use the CLI can be found in the
README. We provide an OPAM remote with
all the working versions of the new libraries and compiler patches required to
drive the new documentation engine.

Future

As previously mentioned, codoc and its
constituent libraries doc-ock-lib
and doc-ock-xml are still under
heavy development and are not yet feature complete. Notably, there are some
important outstanding issues:

We are very happy to take bug reports and patches at
https://github.com/dsheets/codoc/issues. For wider suggestions, comments,
complaints and discussions, please join us on the
Platform mailing list.
We do hope that you'll let us know what you think and help us build a next
generation documentation tool which will serve our community admirably.

The Xapi toolstack is built from a large set of libraries and components
which are
developed independently and versioned separately. It's easy for us to
share code with other open-source projects like
Mirage, however
this flexibility comes
with a cost: when one binary such as "xapi" (the cluster manager)
depends on 45 separate libraries,
how do we quickly set up a
build environment?
Exactly which libraries do we need? How do we apply updates?
If we change one of these libraries (e.g. to make a bugfix), exactly which
bits should we rebuild?
This is where OPAM,
the source package manager, makes everything easy.

Installing a build environment with OPAM is particularly easy.
For example in a CentOS 6.5 VM,
first install OPAM:

and then:

$ opam init --comp=4.01.0
$ eval `opam config env`

Next install the necessary C libraries and development tools for xapi
using a command like

OPAM also makes iterative development very easy.
Consider a scenario where a
common interface has to be changed.
Without OPAM we have to figure out which components to rebuild manually--
this is both time-consuming and error-prone. When we want to make some
local changes we simply clone the repo and tell OPAM to "pin" the package
to the local checkout. OPAM will take care of rebuilding only the
dependent packages:

When a linked set of changes are ready to be pushed, we can make a
single pull request
updating a set of components, which triggers the
travis
integration tests.

Summary

The Xapi toolstack is built from a large set of libraries, independently
versioned and released, many of them shared with other projects
(such as Mirage). The libraries are
easy to build and test separately, but the sheer number of dependencies
makes it difficult to build the whole project -- this is where opam
really shines. OPAM simplifies our day-to-day lives by

automatically rebuilding dependent software when dependencies change

allowing us to share 'development remotes' containing bleeding-edge software
amongst the development team

allowing us to 'release' a co-ordinated set of versions with a git push
and then trigger integration tests via travis

The ICFP conference experience can be a remarkable one for students. Some
great ideas have emerged from random corridor conversations between talks with
the likes of Phil Wadler, or from
rain-soaked discussions with Simon
PJ at
Mikeller, or in my case, from being convinced to write
a book while in
a smoky Tokyo bar. This year, it will be held in the beautiful city of
Vancouver in the fall.

We’re committed to growing the ICFP community, not just in numbers but also in
diversity. The Programming Language Mentoring Workshop
has been at capacity since it started and will run again. For the first time ever,
I am really excited to announce that the Ada Initiative will
also be running an Ally Skills
workshop during the conference.

Sustaining these activities and responsible growth means that we need to reach
ever wider to support the activities of the (not-for-profit) ICFP conference.
So as this year’s industrial relations chair, I wish to invite any organization
that wishes to support ICFP to get in touch with us (e-mail at avsm2@cl.cam.ac.uk) and sponsor us. I’ve put an abridged
version of the e-mail solicitation below that describes the benefits.
Sponsorship can start as low as $500 and is often tax deductible in many
countries.

I’m writing to ask if you would be willing to provide corporate financial
support for the 20th ACM SIGPLAN International Conference on Functional
Programming (ICFP), which takes place in Vancouver, Canada, from August 30th
through September 5th, 2015:

http://icfpconference.org/icfp2015/

Corporate support funds are primarily used to subsidize students – the
lifeblood of our community – and in turn serve to raise the community profile
of the supporting companies through a high-profile industrial recruitment
event.

Last year, unprecedented levels of support from you and folks like you at
over 25 companies and institutions made it possible for students from all
over the world to attend ICFP 2014 in Sweden. The Industrial Reception, open
to all attendees, was by all accounts a roaring success. All 2014 sponsoring
companies had the opportunity to interact with the gathered students,
academics, and software professionals.

This year, let’s build on that success and continue to grow our community, and
bring even more students to ICFP 2015 in Vancouver!

Your generosity will make it possible for students from all over the world to
attend ICFP, the premier conference in functional programming. There, they
will meet luminaries in the field, as well as people who’ve built a successful
career and/or business on functional programming. They will return home
inspired to continue pursuing functional programming in the confidence that
exciting future careers await them. For the first time, we will also host an
Ally Skills workshop by the Ada Foundation, as well as continue the
successful student mentoring workshop from previous years.

This year, we’re continuing similar system of levels of financial support as
last year. Our goal is to enable smaller companies to contribute while allowing
larger companies to be as generous as they wish (with additional benefits, in
recognition of that generosity).

The support levels, and their associated benefits and pledge amounts and
benefits are as follows (costs in US dollars).

Thank you for your time and especially for your generosity! I look forward to
seeing you in Vancouver. If you are willing to be a sponsor, it would be
helpful to hear back by March 9th to help us plan and budget.

If you are interested, please get in touch with me
or any of the organizing committee.
If you’re interested in helping out ICFP in a non-financial capacity (for
example as a student volunteer), then there will also be plenty of opportunity
to sign up later in the year.

FireEye has invented a purpose-built, virtual machine-based security platform that provides real-time threat protection to enterprises and governments worldwide against the next generation of cyber attacks. These highly sophisticated cyber attacks easily circumvent traditional signature-based defenses, such as next-generation firewalls, IPS, anti-virus, and gateways. The FireEye Threat Prevention Platform provides real-time, dynamic threat protection without the use of signatures to protect an organization across the primary threat vectors and across the different stages of an attack life cycle. The core of the FireEye platform is a virtual execution engine, complemented by dynamic threat intelligence, to identify and block cyber attacks in real time. FireEye has over 3,100 customers across 67 countries, including over 200 of the Fortune 500.

Job Description:

In Dresden, Germany, an outstanding team of formal methods engineers uses formal methods tools, such as proof assistants, to develop correctness proofs for FireEye's leading edge products.
In real world applications of formal methods tools, automation is often not sufficient for the specific problems at hand. Therefore, we are seeking outstanding software developers with a passion for implementing both well-designed as well as ad-hoc formal methods software tools for proof refactoring, proof search, systematic testing and other areas.

Responsibilities:

Design and development of software tools, scripts and automated processes for various tasks, supporting formal method tools

Refactoring, redesigning and rewriting earlier ad-hoc solutions in a systematic way

Maintaining existing tools and scripts

Continuous focus and contribution in the areas of performance, reliability, scalability, and maintainability of the product

Active participation in our ongoing process enhancements in software development and verification practices

Desired Skills & Experience

BS or MS in Computer Science or equivalent experience

Strong programming skills in OCaml or Haskell under Linux

Good skills in scripting languages, such as bash, perl or python

Good knowledge of logic and formal logical reasoning

Experience with parallel programs running on multiple machines and cores

is inadmissible because it relies on List.rev and thus is not entirely in terms of fold_right.

Recall that given a function $f$, a seed $e$ and a list $[a_{1}; a_{2}; \cdots; a_{N}]$, fold_left computes $f\;(\cdots f\;(f\;e\;a_{1})\;a_{2}\;\cdots)\;a_{N}$ whereas fold_right computes $f\;a_{1}\;(f\;a_{2}\;(\cdots\;(f\;a_{N}\;e)\cdots))$. There's really no choice but to delay computation and the expression that solves this problem is this.

Jane Street is a proprietary quantitative trading firm, focusing primarily on trading equities and equity derivatives. We use innovative technology, a scientific approach, and a deep understanding of markets to stay successful in our highly competitive field. We operate around the clock and around the globe, employing over 380 people in offices in New York, London and Hong Kong.

The markets in which we trade change rapidly, but our intellectual appro…

Jane Street is a proprietary quantitative trading firm, focusing primarily on trading equities and equity derivatives. We use innovative technology, a scientific approach, and a deep understanding of markets to stay successful in our highly competitive field. We operate around the clock and around the globe, employing over 380 people in offices in New York, London and Hong Kong.

The markets in which we trade change rapidly, but our intellectual approach changes faster still. Every day, we have new problems to solve and new theories to test. Our entrepreneurial culture is driven by our talented team of traders and programmers. At Jane Street, we don't come to work wanting to leave. We come to work excited to test new theories, have thought-provoking discussions, and maybe sneak in a game of ping-pong or two. Keeping our culture casual and our employees happy is of paramount importance to us.

We are looking to hire great software developers with an interest in functional programming. OCaml, a statically typed functional programming with similarities to Haskell, Scheme, Erlang, F# and SML, is our language of choice. We've got the largest team of OCaml developers in any industrial setting, and probably the world's largest OCaml codebase. We use OCaml for running our entire business, supporting everything from research to systems administration to trading systems. If you're interested in seeing how functional programming plays out in the real world, there's no better place.

The atmosphere is informal and intellectual. There is a focus on education, and people learn about software and trading, both through formal classes and on the job. The work is challenging, and you get to see the practical impact of your efforts in quick and dramatic terms. Jane Street is also small enough that people have the freedom to get involved in many different areas of the business. Compensation is highly competitive, and there's a lot of room for growth.

Lookingglass is seeking a qualified Senior Development Engineer to join our team!

Are you an experienced senior software engineer in security, networking, cloud and big data? Are you interested in cyber security or improving the security of the Internet? Do you push yourself to be creative and innovative and expect the same of others?

At Lookingglass, we are driven and passionate about what we do. We believe that teams deliver great products not individuals. We inspire each other and our custo…

Lookingglass is seeking a qualified Senior Development Engineer to join our team!

Are you an experienced senior software engineer in security, networking, cloud and big data? Are you interested in cyber security or improving the security of the Internet? Do you push yourself to be creative and innovative and expect the same of others?

At Lookingglass, we are driven and passionate about what we do. We believe that teams deliver great products not individuals. We inspire each other and our customers every day with technology that improves the security of the Internet and of our customer’s. Behind our success is a team that thrives on collaboration and creativity, delivering meaningful impact to our customers.

Last summer we announced the beta release of a clean-slate implementation of
TLS in pure OCaml, alongside a series of blog posts that described
the libraries and the thinking behind them. It took two hackers six months
— starting on the beach — to get the stack to that point and
their demo server is still going strong. Since then, the team has
continued working and recently presented at the 31st Chaos
Communication Congress.

Last summer we announced the beta release of a clean-slate implementation of
TLS in pure OCaml, alongside a series of blog posts that described
the libraries and the thinking behind them. It took two hackers six months
— starting on the beach — to get the stack to that point and
their demo server is still going strong. Since then, the team has
continued working and recently presented at the 31st Chaos
Communication Congress.

The authors are putting their stack to the test again and this time they've
built a Bitcoin Piñata! Essentially, they've hidden a
private key to a bitcoin address within a Unikernel running on Xen. If you're
able to smash your way in, then you get to keep the spoils.

There's more context around this in my Piñata post and you can see
the details on the site itself. Remember that the codebase is
all open (as well as issues) so there's nothing to
reverse engineer. Have fun!

Last summer we announced the beta release of a clean-slate implementation of
TLS in pure OCaml, alongside a series of blog posts that described
the libraries and the thinking behind them. It took two hackers six months
— starting on the beach — to get the stack to that point and
their demo server is still going strong. Since then, the team has
continued working and recently presented at the 31st Chaos
Communication Congress.

Last summer we announced the beta release of a clean-slate implementation of
TLS in pure OCaml, alongside a series of blog posts that described
the libraries and the thinking behind them. It took two hackers six months
— starting on the beach — to get the stack to that point and
their demo server is still going strong. Since then, the team has
continued working and recently presented at the 31st Chaos
Communication Congress.

The latest example goes quite a bit further than a server that just displays
the handshake. This time, the team have constructed a Xen unikernel that’s
holding a private key to a bitcoin address and are asking people to try and
break in. Hence, they’ve called it the Bitcoin Piñata!*

What the Bitcoin Piñata does

The Piñata unikernel will transmit its private bitcoin key if you can
successfully set up a TLS connection but it’s rigged so that it will only
create that connection if you can present the certificate it’s expecting to
see — which has been signed appropriately. Of course, you’re not being given
the secret key with which to do that signing and that means there should be
no way for anyone to form a TLS connection with the Piñata.
In order to get the private key to the bitcoin address, you’ll have to smash
your way in.

Helpfully (perhaps), things are set up so that you can make the Piñata talk
to itself, allowing you to eavesdrop on a successful connection and
see the encrypted traffic. In addition, all the code and libraries are
open-source so you can look through any of the codebase. There isn’t anything
that anyone will have to reverse engineer, which should make this a little
more enjoyable.

This contest is set to run until mid-March or whenever the coins are taken.
If someone does manage to get in, please do let us know how!

The Rubber-hose approach

Of course there are many other ways to get at the private key and as many
people like to comment, the human element is sometimes the weakest link —
after all, a safe is only as secure as the person with the combination.

In this case, there is obviously a secret key or certificate somewhere
that could be presented so it may be tempting to go hunting for that. Perhaps
phishing attempts on the authors may yield a way forward, or maybe just
straight-forward Rubber-hose cryptanalysis! Sure, these
options might provide a result† but this is meant to be fun.
The authors haven’t specified any rules but please be nice and focus on the
technical things around the Piñata‡. Don’t be this guy.

What’s the point of this contest?

Even though the Bitcoin Piñata is clearly a contest, nobody is deluding
themselves into thinking that if the coins are still there in March, that
somehow the stack can be declared ‘undefeated’ — while pleasing, that
result wouldn’t necessarily prove anything. Contests have their place but as
Bruce Schneier already pointed out, they are not useful mechanisms
to judge security.

However, it does give us the chance to engage in some shameless self-promotion
and try to draw vast amounts of attention to the work. That, and the chance to
stress-test the stack in the wild. Ultimately, we want to use this code in
production but must take a lot of care to get there and want to be sure that
it can bear up. This is just one more way of learning what happens when
putting something ‘real’ out there.

If the Bitcoins do end up being taken, then there’s definitely something
valuable that the team can learn from that. Regardless of the Piñata, if we
have more people exploring the TLS codebase or trying it out for
themselves, it will undoubtedly be A Good Thing.

Responsible sidenote

For clarity and to avoid any doubt, please be aware that the TLS codebase is
missing external code audits and is not yet intended for use in any security
critical applications. All development is done in the open, including the
tracking of security-related issues, so please do consider
auditing the code, testing it in your services and reporting issues.

* If you've never come across a piñata before, hopefully
the gif in the post gives you an idea. If not, the
wiki page
will surely help, where I learned that the origin may be Chinese rather
than Spanish!

† Of course, I'm not suggesting that
anyone would actually go this far. I'm simply acknowledging that there is
a human factor and asking that we put it aside.

‡ Edit to add: After seeing
Andrea's tweet I should point out that any part of
MirageOS, including the networking stack, OCaml runtime etc is a
legitimate vector. It's why there's a
manifest of the libraries that have been used to build the Piñata!

Earlier this week, I had a need for a recursive list, that is, a list defined in terms of itself. I think, "back in the day" implementing a data structure of that sort would have been a snap for the everyday C programmer. Today, in this modern C++ world I found myself struggling a little and came to think that maybe the old ways are fading :)

For motivation, here's a couple of examples of the sort of thing I'm talking about.

(1) The list [0; 1; 0; 1; 0; 1; ...] is a list with a cycle in it. In O…

Earlier this week, I had a need for a recursive list, that is, a list defined in terms of itself. I think, "back in the day" implementing a data structure of that sort would have been a snap for the everyday C programmer. Today, in this modern C++ world I found myself struggling a little and came to think that maybe the old ways are fading :)

For motivation, here's a couple of examples of the sort of thing I'm talking about.

Last weekend was spent at one of the world’s biggest open source conferences,
FOSDEM. You can check out last year’s review to get an idea of
the scale of the event. Since there’s no registration process, it’s difficult
to estimate how many people attend but given how many rooms there are, and how
full they are, it’s easily several thousand. I was impressed last year at how
smoothly things went and the same was true this year.

Last weekend was spent at one of the world’s biggest open source conferences,
FOSDEM. You can check out last year’s review to get an idea of
the scale of the event. Since there’s no registration process, it’s difficult
to estimate how many people attend but given how many rooms there are, and how
full they are, it’s easily several thousand. I was impressed last year at how
smoothly things went and the same was true this year.

The main reason to attend this time was to run a demo of MirageOS from an ARM
board — one of the main advances since the previous conference. I looked over
all the things we’d achieved since last year and put together a demo that
showcases some of the capabilities as well as being fun.

2048 from a Unikernel on an ARM board

The demo was to serve the 2048 game from a Unikernel running on a Cubieboard2
with its own access point. When people join the wifi network, they get
served a static page and can begin playing the game immediately.

Code for making a static website — Since the game is completely
self-contained (one html file and one js file). I only need to convert a static
website into a unikernel. That’s trivial and
many people have done it before.

A Cubieboard with a wifi access point — There are pre-built images on the
MirageOS website, which make part of this easy. However, getting the
wifi access point up involves a few more steps.

The first two pieces should be straightforward and indeed, I had a working
unikernel serving the 2048 game within minutes (unix version on my laptop).
The additional factors around the ARM deployment is where things were a little
more involved. Although this was technically straightforward to set up, it
still took a while to get all the pieces together. A more detailed
description of the steps is in my fosdemo repository and in
essence, it revolves around configuring the wifi access point and setting up a
bridge (thanks to Mindy, Magnus and David for getting this
working).

Once this was all up and running, it was a simple matter to configure the
board to boot the unikernel on startup, so that no manual intervention would
be required to set things up at the booth.

Running the demo

I gave the demo at the Xen booth and it went very well. There was a small
crowd throughout my time at the booth I’m convinced that the draw of a board
with glowing LEDs should not be underestimated. Many people we’re happy to
connect to the access point and download the game to their browser but there
were two main things I learnt.

Firstly, demos involving games will work if people actually know the game.
This is obvious, but I’d assumed that most people had already played 2048 —
especially the crowd I’d expect to meet at FOSDEM. It turned out that around
a third of people had no idea what to do when the game loaded onto their
browser. They stared blankly at it and then blankly at me. Of course, it was
trivial to get them started and they were soon absorbed by it — but it still
felt like some of the ‘cool-factor’ had been lost.

The second thing was that I tried to explain too much to people in much too
short a time. This particular demo involved Xen unikernels, js_of_ocaml and a
Cubieboard2 with a wifi access point.
There’s a surprisingly large amount of technology there, which
is difficult explain to a complete newcomer within one or two minutes. When
it was obvious someone hadn’t heard of unikernels, I focused on the approach
of library operating systems and the benefits that Mirage brings. If a visitor
was already familiar with the concept of unikernels, I could describe the rest
of the demo in more detail.

Everything else did go well and next time I’d like to have a demo like this
running with Jitsu. That way, I could configure it so that a unikernel
would spin up, serve the static page and then spin down again. If we can
figure out the timing, then providing stats in the page about the lifetime of
that unikernel would also be great, but that’s for another time.

Sidenote: The beginnings of a ‘personal cloud’

One of the things we’re keen to work towards is the idea of
personal clouds. It’s not a stretch to imagine that a Cubieboard2,
running the appropriate software, could act as one particular node in a
network of your own devices. In this instance it’s just hosting a fun and
simple game but more complex applications are also possible.

Huge range of sessions and talks

Of course, there was lots more going on than just my demo and I had a great
time attending the talks. Some in particular that stood out to me were those
in the open source design room, which was a new addition this year. It
was great to learn that there are design people out there who would like to
contribute to open source (get in touch, if that’s you!). I also had a
chance to meet (and thank!) Mike McQuaid in his Homebrew talk.
FOSDEM is one of those great events where you can meet in person all those
folks you’ve only interacted with online.

Overall, it was a great trip and I thoroughly recommend it if you’ve never
been before!

A few years back I taught a course on Paradigms of Programming Languages that spent most of its time in two particular paradigms: functional programming and logic programming. For the functional programming section I decided to use OCaml because I thought it would be less of a culture shock to the students than using Haskell (also because I really like OCaml).

The course went well, and after that I began thinking about writing a more organized set of notes or even a book about the language in P…

A few years back I taught a course on Paradigms of Programming Languages that spent most of its time in two particular paradigms: functional programming and logic programming. For the functional programming section I decided to use OCaml because I thought it would be less of a culture shock to the students than using Haskell (also because I really like OCaml).

The course went well, and after that I began thinking about writing a more organized set of notes or even a book about the language in Portuguese. There is next to nothing about OCaml in Portuguese (as far as I know), and though I think it’s essential for students to know how to read English, I also think it would be ideal if more content were available in our language. It’d also be useful to give to students who want to work with me in projects that use OCaml.

Last year I dug up the old notes and started writing it. Despite its origins, it’s not a textbook. The publisher, Casa do Código, sells books for developers (similar to O’Reilly and Manning), so it’s more oriented to getting developers up to speed with functional programming than with the theoretical aspects of the paradigm. It still could be used as supplemental reading in courses like Functional Programming or Programming Language Paradigms.

More about the book (in Portuguese) and additional resources can be found at my site. The book can be bought (ebook, print book, or both) at the publisher’s site.

We'll be meeting in the Computer Lab next Tuesday (10th February 2015) for another evening of compiler hacking. All welcome! Please add yourself to the Doodle poll if you're planning to come along, and sign up to the mailing list to receive updates.

Talk: Generating code with polymorphic let (Oleg Kiselyov)

Generating code with polymorphic let

We'll be meeting in the Computer Lab next Tuesday (10th February 2015) for another evening of compiler hacking. All welcome! Please add yourself to the Doodle poll if you're planning to come along, and sign up to the mailing list to receive updates.

Talk: Generating code with polymorphic let (Oleg Kiselyov)

Generating code with polymorphic let

One of the simplest ways of implementing staging is source-to-source
translation from the quotation-unquotation code to code-generating
combinators. For example, MetaOCaml could be implemented as a
pre-processor to the ordinary OCaml. However simple, the approach is
surprising productive and extensible, as Lightweight Modular Staging
(LMS) in Scala has demonstrated. However, there is a fatal flaw:
handling quotations that contain polymorphic let. The translation to
code-generating combinators represents a future-stage let-binding with
the present-staging lambda-binding, which is monomorphic. Even if
polymorphic lambda-bindings are allowed, they require type
annotations, which precludes the source-to-source translation.

We show the solution to the problem, using a different translation. It
works with the current OCaml. It also almost works in theory,
requiring a small extension to the relaxed value
restriction. Surprisingly, this extension seems to be exactly the one
needed to make the value restriction sound in a staged language with
reference cells and cross-stage-persistence.

The old, seems completely settled question of value restriction is
thrown deep-open in staged languages. We gain a profound problem to
work on.

This book presents 30 algorithm problems and their functional solutions. The reason that they are called as pearls is what I quote below:

Just as natural pearls grow from grains of sand that have irritated oysters, these programming pearls have grown from real problems that have irritated programmers. The programs are fun, and they teach important programming techniques and fundamental design principles.

These pearls are all classic and togehter with the well presented functional approaches, they become very helpful for ones who really wish to sharpen their functional programming on algorithms. I have so far solved / fully understood 17 pearls and hmm...the rest are indeed difficult, at least for me. Nevertheless, I would like to share my journey of studying this book with you via a series of posts, each of which is a pearl in the book. All the posts will differ from the book in the following ways:

Only OCaml is used for the implementations (for the obvious reason: I am a fan of OCaml)

More general (functional) analysis techniques are adopted. The book heavily focuses on algebraic approach, namely, design by calculation, which honestly is too much for me. I mean I can understand the methodology, but cannot master it. So during my study, I did not replicate the algebraic thinking; instead, I tried to sniff around the functional sprit behind the curtain and consider the algebraic part as the source of hints.

Tail-recursive implementation will be provided if possible. The haskell implementations in the book do not consider much about tail-recursive because a) haskell is lazy; b) haskell's compiler is doing clever things to help recursion not blow. OCaml is different. It is not lazy and does not rely on the compiler to do optimisations on the potential stackoverflow. So as OCaml developers, we need to care about tail-recursive explicitly.

Ok. Let's get started.

The Min Missing Natural Number

Given a unordered list of distinct natural numbers, find out the minimum natural number that is not in the list.

For example, if the list is [8; 2; 3; 0; 12; 4; 1; 6], then 5 is the minimum natural number that is missing.

O(n) solution is desired.

Analysis of the problem description

The description of an algorithm problem specifies the input, the output and the constraint. Yet, it is more than just telling us what to achieve. Most of the time, the literatures can provide us hints for possible solutions. Let's break down the description first:

unordered list

distinct

natural numbers

minimum and missing

unordered list

The input list is not sorted. If this is specified explicitly, it implies that ordering is important here. In other words, if the list was sorted, then the problem would not be a problem any more or at least much easier to solve.

distinct

There are no duplicates inside.

natural numbers

All numbers are non-negative integers, i.e., 0, 1, 2, ... This puts a lower boundary on the possible numbers in the input list.

minimum and missing

There might be unlimited numbers not in the input list, but we just need to find the smallest one. When our goal is to locate something with certain characteristic, it would normally be a selection problem. Moreover, for problems related min or max, they normally can be solved by somehow bringing sorting; however, sorting will heavily involve moving all numbers around. As our target is only one number, sorting can be an overkill.

An easy but not optimal solution

From the analysis above, it is obvious that sorting can solve our problem quite easily. We can simply

Have we solved the problem? Let's have a look. The solution above can achieve the goal indeed. However, the time complexity of this solution is O(nlogn) since the sorting bit is dominating, which is worse than the required O(n).

We have to try harder.

Do we need complete order?

So, if we do a sorting, we can obtain the answer but it is too slow. Let's have a look again at what we get after sorting.

If the list is sorted, then it provides us a chance where we check the consecutiveness of the numbers and the first gap is what we want. There are two questions though:

Q1. Shall we care about the consecutiveness after 6?

The answer is no. Since we are chasing for the minimum, i.e., the first missing one, the order of the numbers after the gap doesn't matter any more.

For example, even if 12 is before 8, the result won't be affected.

Q2. Is the order of all the numbers before the gap important?

Let's randomly mess up the order of 0, 1, 2, and 3 a little:

It seems fine as the messed order of those 4 numbers does not affect the position of 4 and 6. But hang on a minute, something is not right there.

We replied on the consecutiveness of 0, 1, 2, and 3 to locate the first gap, and the consecutiveness can be checked via the numbers being sorted. Hence, if the order before the gap was not maintained, how could we scan for consecutiveness and find the gap in the first place? It sounds like a chicken and egg thing.

So can we check for the consecutiveness of the numbers without sorting them?

Hints hidden in "distinct natural numbers"

Yes, we can, and now it is the time to ask for help from distinct natural numbers.

As we described before, natural numbers are integers euqal to or larger than 0. This lower bound 0 plus the constraint of no duplicates gives us the opportunity to check for consecutiveness without requiring all numbers being sorted. Let's first see a perfect consecutive sequnce (starting from 0) of natural numbers:

They are sorted of course. Is there any other characteristic? Or say, for a number inside the sequence, how many other numbers are less than it (on its left side)?

For number 4, there will exact 4 natural numbers less than itself. The same thing will apply on any numbers as long as all those belong to a perfect consecutive sequence starting from 0.

This is also a two-way mapping, i.e., if we are told that there are 4 numbers less than 4 and all of them are before 4, we can be sure that all five numbers can form a consecutive sequence. Most importantly, now whether all numbers are in order or not does not matter any more.

What does a perfect consecutiveness imply?

It implies that among the sequence, no one is missing and if anyone is missing, it must be on the right side of the max of the sequence.

What if for a number, the number of smaller ones does not match its own value?

It means the sequence up to the number won't be consecutive, which implies that there must be at least one a natural number missing, right?

Now we have the weapon we want. In order to check consecutiveness, or say, to know the region where the min missing natural number is, we don't need complete sorting any more. Instead, we just to

pick a number x from the sequence

put all other smaller numbers to its left and count how many are those in num_less

put all larger ones to its right

If num_less = x, then it means x's left branch is perfect and the missing one must be in its right branch. We repeat the whole process in the sequence of right hand side.

Otherwise, we repeat in the left branch. Note that due to distinct, it is impossible num_less > x.

In this way, we cannot identify the min missing one in one go, but we can narrow down the range where it might be through each iteration and eventually we will find it when no range can be narrowed any more.

Sounds like a good plan? let's try to implement it.

Partition and Implementation

Do you feel familiar with the process we presented above? It is actually a classic partition step which is the key part of quicksortwe talked about.

Of course, in this problem, it won't be a standard partition as we need to record num_less. The code should be easy enough to write:

Summary

This is the first pearl and it is not that difficult, especially if you knew quickselect algorithm before.

Actually, many of the pearls involve certain fundamental algorithms and what one need to do is to peel the layers out one by one and eventually solve it via the essentials.

I am making a list now to map the pearls to their according core inside, such as essential data structures, algorithms, patterns, typical solutions, and techniques. And the list will be updated after each pearl.

We spend a lot of time and effort on training new people, and it never stops for long. Right now our winter-intern class is ending; in five months we'll have a slew of new interns to get up to speed, and a few months after that we'll have an incoming class of new hires.

A big part of our new-hire training is OCaml Bootcamp, a month-long immersion program for non-dev hires (mostly trading, but some from other areas like research, systems, and accounting). We don't think everyone at Jane Street sh…

We spend a lot of time and effort on training new people, and it never stops for long. Right now our winter-intern class is ending; in five months we'll have a slew of new interns to get up to speed, and a few months after that we'll have an incoming class of new hires.

A big part of our new-hire training is OCaml Bootcamp, a month-long immersion program for non-dev hires (mostly trading, but some from other areas like research, systems, and accounting). We don't think everyone at Jane Street should be a full-on software developer, but writing code is such a useful way to get things done that it's worth teaching the basics to a broad set of people.

Teaching programming, especially to people who are not planning on becoming software developers, is an opportunity to reflect on how unnecessarily hard programming can be. There's a huge learning curve as you struggle to learn your way around the Unix command-line, or figure out the key commands for controlling a 1970's era text editor like Emacs or Vi, or puzzle through the semantics of a distributed version control system like Mercurial. And all of that is before you even start writing code!

To me, this serves as a reminder of the importance of good tools. The quality of your tools can increase or decrease the steepness of the learning curve, and they also affect your day-to-day efficiency after you've climbed up that hill.

Tools are easy to undervalue. Most of our development time is spent, as it should be, on our first order problems -- writing the code that powers the systems that let us trade. And the connection between better tools and better trading can seem tenuous.

But really it's not tenuous at all. If you spend all your time on first order problems, you'll discover you're not solving them as fast as you should be. Getting things done effectively requires optimizing your own productivity, and to do that, you have to spend some time sharpening your tools.

And we've done a fair bit of sharpening. One recent example is Iron, a new code review and release management system that we started using last summer. Last year, we also rolled out a new build system called Jenga, which greatly simplified and sped up the process of compiling our code. Plus, we switched to a new version of OCaml, which includes a big set of improvements, some of which were specifically aimed at improving our development process [7]. And we funded some former interns to improve Merlin, a fantastic tool that provides IDE-like features like context-sensitive autocompletion in a way that can be easily integrated into multiple editors.

Jane Street is a pretty small shop --- we have fewer than 65 full time developers --- but even at our modest scale, spending time on tools is a big win. But it's really about more than dev tools. Thinking about how to make the people around you more effective informs how you work more generally, changing how you design libraries, how you manage services, and how (and whether!) you write documentation.

And in addition to making for a more effective organization, it's also a more pleasant way to live.

we provide a private access to our internal repositories for academia users and our clients.

Quick Evaluation

A quick comparison between this new version and the previous releases is given below. Timeout is set to 60 seconds. The benchmark is made of 19044 formulas: (a) some of these formulas are known to be invalid, and (b) some of them are out of scope of current SMT solvers. The results are obtained with Alt-Ergo's native input language.

public release0.95.2

public release0.99.1

private release1.00

Proved Valid

15980

16334

17638

Proved Valid (%)

84,01 %

85,77 %

92,62 %

Required time (seconds)

10831

10504

9767

Average speed (valid formulas per second)

1,47

1,55

1,81

Main Novelties of Alt-Ergo 1.00

General Improvements

theories data structures: semantic values (internal theories representation of terms) are now hash-consed. This enables the use of hash-based comparison (instead of structural comparison) when possible

theories combination: the dispatcher component, that sends literals assumed by the SAT solver to different theories depending on whether these literals are equalities, disequalities or inequalities, has been re-implemented. The new code is much more simpler and enables some optimizations and factorizations that could not be made before

case-split analysis: we made several improvements in the heuristics of the case-split analysis mechanism over finite domains

explanations propagation: we improved explanations propagation in congruence closure and linear arithmetic algorithms. This makes the proofs faster thanks to a better back-jumping in the SAT solver part

linear integer arithmetic: we re-implemented several parts of linear arithmetic and introduced important improvements in the Fourier-Motzkin algorithm to make it run on smaller sub-problems and avoid some useless executions. These optimizations allowed a significant speed up on our internal benchmarks

data structures: we optimized hash-consing and some functions in the "formula" and "literal" modules

SAT solving: we made a lot of improvements in the default SAT-solver and in the SatML plugin. In particular, the solvers now send lists of facts (literals) to "the decision procedure part" instead of sending them one by one. This avoids intermediate calls to some "expensive" algorithms, such as Fourier-Motzkin

Matching: we extended the E-matching algorithm to also perform matching modulo the theory of records. In addition, we simplified matching heuristics and optimized the E-matching process to avoid computing the same instances several times

Memory management: thanks to the ocp-memprof tool (http://memprof.typerex.org/), we identified some parts of Alt-Ergo that needed some improvements in order to avoid useless memory allocations, and thus unburden the OCaml garbage collector

the function that retrieves the used axioms and predicates (when option 'save-used-context' is activated) has been improved

Bug Fixes

6 in the "inequalities" module of linear arithmetic

4 in the "formula" module

3 in the "ty" module used for types representation and manipulation

2 in the "theories front-end" module that interacts with the SAT solvers

1 in the "congruence closure" algorithm

1 in "existential quantifiers elimination" module

1 in the "type-checker"

1 in the "AC theory" of associative and commutative function symbols

1 in the "union-find" module

New OCamlPro Plugins

profiling plugin: when activated, this plugin records and prints some information about the current execution of Alt-Ergo every 'x' seconds: In particular, one can observe a module being activated, a function being called, the amount of time spent in every module/function, the current decision/instantiation level, the number of decisions/instantiations that have been made so far, the number of case-splits, of boolean/theory conflicts, of assumptions in the decision procedure, of generated instances per axiom, ....

fm-simplex plugin: when activated, this plugin is used instead of the Fourier-Motzkin method to infer bounds for linear integer arithmetic affine forms (which are used in the case-split analysis process). This module uses the Simplex algorithm to simulate particular runs of Fourier-Motzkin, which makes it scale better on linear integer arithmetic problems containing a lot of inequalities

New Options

-version-info: prints some information about this version of Alt-Ergo (release and compilation dates, release commit ID)

-no-theory: deactivate theory reasoning. In this case, only the SAT-solver and the matching parts are working

-inequalities-plugin: specify a plugin to use, instead of the "default" Fourier-Motzkin algorithm, to handle inequalities of linear arithmetic

-tighten-vars: when this option is set, the Fm-Simplex plugin will try to infer bounds for integer variables as well. Note that this option may be very expensive

-profiling-plugin: specify a profiling plugin to use to monitor an execution of Alt-Ergo

How and why I designed my own approximated implementations
of log, exp, sin and cos. The approximation error is quite
small, and the functions are fast and vectorizable. For example, my
logapprox function is 7.5x faster in tight loops than
the icsi_log
function, while being more precise.

Recently, I helped my wife optimizing some code doing simulations for her bio-statistics research project. Without going into details, her problem requires doing Bayesian inference using MCMC methods. T…

How and why I designed my own approximated implementations
of log, exp, sin and cos. The approximation error is quite
small, and the functions are fast and vectorizable. For example, my
logapprox function is 7.5x faster in tight loops than
the icsi_log
function, while being more precise.

Recently, I helped my wife optimizing some code doing simulations for her bio-statistics research project. Without going into details, her problem requires doing Bayesian inference using MCMC methods. Those methods are typically very CPU intensive, and in this particular code, most of the time was spent computing logarithms and exponentials. This is a bit disappointing, because the results of these computations are in the end only used as a probability from which we are going to draw a Boolean (they are rejection probabilities of the Metropolis-Hastings algorithm). We are clearly spending a lot of time computing these number with the full double precision, while we need much less.

My first optimization idea was to use an approximated version of the logarithm and exponential functions. Namely, I used the well-known icsi_log function for the logarithm, and I have written a similar function for exponential. I am going to explain the principle for logarithm: exponential use similar tricks, but in the other direction. The first idea is to transform the problem of natural logarithm into base-2 logarithm by multiplying the result by \(\log 2\). Next, recall that floating-point numbers are represented by a mantissa and an exponent: using bitmasks, it is very easy to extract from a normalized floating point number \(f\) two numbers \(m\) and \(e\) such that \(f = m2^e\), \(m \in [1,2[\), \(e \in \mathbb{Z}\). Thus, we have \(\log_2 f = \log_2 m + e\), so that we reduced the problem into computing the logarithm of a number in \([1,2[\), which is done in icsi_log using precomputed look-up tables.

While the use of these functions brought some speedup, there were two major drawbacks: first, because the look-up table must not be too big (otherwise it will not fit in low-level cache, and look-up would be too slow), the precision is limited. Second, the loops that contained calls to the approximated functions contained accesses to arrays with random indexes, and thus cannot be vectorized.

Loop vectorization is a technique able to provide huge speedups (typically x3 at least), when the hottest part of the code is simple enough. Not-so-old processors have special instructions, called SIMD instructions, that are able to execute the same operation on several pieces of data simultaneously. In the x86 world, they are called SSE instructions. They are typically used when doing repetitive operations on arrays, and can lead to big speedups. Until recently, if you wanted your code to be vectorized, you had to manually write code that explicitly used these features of the processor. This leads to code that is neither easily maintainable nor portable. However, a lot of progress had been done, and recent compilers are able to vectorize automatically simple loops: that is, you write loops as if they were completely sequential, and the compiler is able to figure out the iterations are independent, and, when possible, bind them together by using SIMD instructions. This can lead to huge speedups, and I wanted to take advantage of this opportunity. Automatic vectorization is enabled in recent versions of GCC, when using -O3 option. For some loops, it is necessary to add -unsafe-math-optimizations to allow GCC to reorder some operations and enable vectorization. The problem with icsi_log is that accessing tabulated values cannot be vectorized, and so these functions blocked the automatic vectorization process of GCC.

I searched for other approximated approximations of these functions: I found Intel's Approximate Math Library, which is oldish, written in assembly, thus not portable for other platforms, and needed rewriting if used for GCC. Moreover, its use would need manual vectorization, which is not handy.

So, I crafted my own version of log and exp, reusing the trick about the binary representation of floating-point numbers, but replacing the table look-up by a good polynomial approximation. I used the Sollya tool to find those polynomial approximations. Here is, for example, the Sollya script that can be used to find the best polynomial approximation of the natural logarithm in the interval \([1, 2]\) -- using the Remez algorithm. It also plots the error :

f = remez(log(x), 4, [1,2]);
plot(f-log(x), [1,2]);

plot of the approximation error

These functions are vectorizable, giving big speedups if used in appropriate loops. We gain x21 for exponentials (compared to the expf function in glibc, which does not give much more precision) and x38 for logarithms (compared to logf function in glibc, which gives almost twice more correct mantissa bits). Moreover, their precision is quite good: for exponential, the relative error is bounded by \(10^{-5}\), and for logarithm, the absolute error is bounded by \(7.10^{-5}\) (both bounds stand for "reasonable" inputs are given, that is when denormal numbers are not involved). That's better than both tabulated and Intel's versions.

In the hope it could be useful to someone, I provide the implementations of both functions, logapprox and expapprox, with their benchmarking scripts (feel free to reproduce this), under MIT license: https://github.com/jhjourdan/SIMD-math-prims

I also provide the implementation (sinapprox, cosapprox) of approximated \(\sin\) and \(\cos\), based on polynomial approximations, and correct only in \([-\pi, \pi]\), which represents a common usecase. Future work would be to design approximations for other common math functions: I'd be happy to implement or merge your own, if asked for.

When I wrote the section of When we need later substitution in Mutable, I struggled. I found out that I didn't fully understand the recursive memoize myself, so what I had to do was just copying the knowledge from Real World OCaml. Luckily, after the post was published, glacialthinkercommented in reddit:

(I never thought before that a recursive function can be split like this, honestly. I don't know how to induct such a way and can't explain more. I guess we just learn it as it is a…

When I wrote the section of When we need later substitution in Mutable, I struggled. I found out that I didn't fully understand the recursive memoize myself, so what I had to do was just copying the knowledge from Real World OCaml. Luckily, after the post was published, glacialthinkercommented in reddit:

(I never thought before that a recursive function can be split like this, honestly. I don't know how to induct such a way and can't explain more. I guess we just learn it as it is and continue. More descriptions of it is in the book.)

This is "untying the recursive knot". And I thought I might find a nice wikipedia or similiar entry... but I mostly find Harrop. :) He actually had a nice article on this many years back in his OCaml Journal. Anyway, if the author swings by, searching for that phrase may turn up more material on the technique.

It greatly enlightened me. Hence, in this post, I will share with you my futher understanding on recursive memoize together with the key cure untying the recursive knot that makes it possible.

Simple Memoize revamped

We talked about the simple memoize before. It takes a non-recursive function and returns a new function which has exactly the same logic as the original function but with new new ability of caching the argument, result pairs.

The greatness of memoize is its flexibility: as long as f takes a single argument, memoize can make a memo version out of it without touching anything inside f.

This means while we create f, we don't need to worry about the ability of caching but just focus on its own correct logic. After we finish f, we simply let memoize do its job. Memoization and functionality are perfectly separated.

Unfortunately, the simple memoize cannot handle recursive functions. If we try to do memoize f_rec, we will get this:

f_rec is a recursive function so it will call itself inside its body. memoize f_rec will produce f_rec_memo which is a little similar as the previous f_memo, yet with the difference that it is not a simple single call of f_rec arg like we did f arg. Instead, f_rec arg may call f_rec again and again with new arguments.

Let's look at it more closely with an example. Say, arg in the recursive process will be always decreased by 1 until 0.

Let's first od f_rec_memo 4.

f_rec_memo will check the 4 against Hashtbl and it is not in.

So f_rec 4 will be called for the first time.

Then f_rec 3, f_rec 2, f_rec 1 and f_rec 0.

After the 5 calls, result is obtained. Then 4, result pair is stored in Hashtbl and returned.

Now let's do f_rec_memo 3, what will happen? Obviously, 3 won't be found in Hashtbl as only 4 is stored in step 5.

But should 3, result pair be found? Yes, it should of course because we have dealt with 3 in step 4, right?

Why 3 has been done but is not stored?

ahh, it is because we did f_rec 3 but not f_rec_memo 3 while only the latter one has the caching ability.

Thus, we can use memoize f_rec to produce a memoized version out of f_rec anyway, but it changes only the surface not the f_rec inside, hence not that useful. How can we make it better then?

Recursive Memoize revamped

What we really want for memoizing a recursive function is to blend the memo ability deep inside, like this:

Essentially we have to replace f_rec inside with f_rec_memo:

And only in this way, f_rec can be fully memoized. However, we have one problem: **it seems that we have to change the internal of f_rec.

If we can modify f_rec directly, we can solve it easily . For instance of fibonacci:

In the above solution, we replaced the original fib_rec inside with fib_rec_memo, however, we also changed the declaration to fib_rec_memo completely. In fact, now fib_rec is totally ditched and fib_rec_memo is a new function that blends the logic of memoize with the logic of fib_rec.

Well, yes, fib_rec_memo_trivial can achieve our goal, but only for fib_rec specificly. If we need to make a memoized version for another recursive function, then we need to change the body of that function again. This is not what we want. We wish for a memoize_rec that can turn f_rec directly into a memoized version, just like what the simple memoize can do for f.

So we don't have a shortcut. Here is what we need to achieve:

we have to replace the f_rec inside the body of f_rec with f_rec_memo

We have keep the declaration of f_rec.

We must assume we can't know the specific logic inside f_rec.

It sounds a bit hard. It is like giving you a compiled function without source code and asking you to modify its content. And more imporantly, your solution must be generalised.

Fortunately, we have a great solution to create our memoize_rec without any hacking or reverse-engineering and untying the recursive knot is the key.

Untying the Recursive Knot

To me, this term sounds quite fancy. In fact, I never heard of it before 2015-01-21. After I digged a little bit about it, I found it actually quite simple but very useful (this recursive memoize case is an ideal demonstration). Let's have a look at what it is.

Every recursive function somehow follows a similar pattern where it calls itself inside its body:

Once a recursive function application starts, it is out of our hands and we know it will continue and continue by calling itself until the STOP condition is satisfied. What if the users of our recursive function need some more control even after it gets started?

For example, say we provide our users fib_rec without source code, what if the users want to print out the detailed trace of each iteration? They are not able unless they ask us for the source code and make a new version with printing. It is not that convenient.

So if we don't want to give out our source code, somehow we need to reform our fib_rec a little bit and give the space to our users to insert whatever they want for each iteration.

Have a look at the above fib_rec carefully again, we can see that the logic of fib_rec is already determined during the binding, it is the fib_recs inside that control the iteration. What if we rename the fib_recs within the body to be f and add it as an argument?

let fib_norec f n =
if n <= 1 then 1
else f (n-1) + f (n-2)
(* we actually should now change the name of fib_norec
to something like fib_alike_norec as it is not necessarily
doing fibonacci anymore, depending on f *)

So now fib_norec won't automatically repeat unless f tells it to. Moreover, fib_norec becomes a pattern which returns 1 when n is <= 1 otherwise addf (n-1) and f (n-2). As long as you think this pattern is useful for you, you can inject your own logic into it by providing your own f.

Going back to the printing requirement, a user can now build its own version of fib_rec_with_trace like this:

Untying the recusive knot is a functional design pattern. It turns the recursive part inside the body into a new parameter f. In this way, it breaks the iteration and turns a recursive function into a pattern where new or additional logic can be injected into via f.

It is very easy to untie the knots for recusive functions. You just give an addition parameter f and replace f_rec everywhere inside with it. For example, for quicksort:

There are more examples in Martin's blog, though they are not in OCaml. A formalized description of this topic is in the article Tricks with recursion: knots, modules and polymorphism from The OCaml Journal.

Now let's come back to recursive memoize problem with our new weapon.

Solve Recursive Memoize

At first, we can require that every recursive function f_rec must be supplied to memoize_rec in the untied form f_norec. This is not a harsh requirement since it is easy to transform f_rec to f_norec.

Once we get f_norec, we of course cannot apply memoize (non-rec version) on it directly because f_norec now takes two parameters: f and arg.

Although we can create f_rec in the way of let rec f_rec arg = f_norec f_rec arg, we won't do it that straightforward here as it makes no sense to have an exactly the same recursive function. Instead, we can for now do something like let f_rec_tmp arg = f_norec f arg.

We still do not know what f will be, but f_rec_tmp is non-recursive and we can apply memoize on it: let f_rec_tmp_memo = memoize f_tmp.

f_rec_tmp_memo now have the logic of f_norec and the ability of memoization. If f can be f_rec_tmp_memo, then our problem is solved. This is because f is inside f_norec controlling the iteration and we wished it to be memoized.

The magic that can help us here is making f mutable. Because f needs to be known in prior and f_rec_tmp_memo is created after, we can temporarily define f as a trivial function and later on after we create f_rec_tmp_memo, we then change f to f_rec_tmp_memo.

While working on an OCaml app to run in the iPhone Simulator, I
discovered another wrapper code construct that looks plausible but is
incorrect. I was able to reproduce the error in a small example under OS
X, so I am writing it up for the record.

The error is in code that calls from Objective C into OCaml. In an OCaml
iOS app these calls happen all the time, since events originate in iOS.
You can imagine events being formed originally from an Objective C-like
substance, then …

While working on an OCaml app to run in the iPhone Simulator, I
discovered another wrapper code construct that looks plausible but is
incorrect. I was able to reproduce the error in a small example under OS
X, so I am writing it up for the record.

The error is in code that calls from Objective C into OCaml. In an OCaml
iOS app these calls happen all the time, since events originate in iOS.
You can imagine events being formed originally from an Objective C-like
substance, then being remanufactured into an OCaml material and passed
through the interface.

As a teensy example, assume that you want to create a point and a
rectangle in C and pass them to a function in OCaml. To make it
interesting, assume that you want to count the fraction of time a
rectangle with randomly chosen origin and size (uniform values in [0,
1]) contains the point (0.5, 0.5).

You, reader, are probably way ahead of me as usual, but the problem is
in this line:

isinside = caml_callback2(*inside, point, ran_rect());

The problem is that ran_rect() allocates OCaml memory to hold the
rectangle and its float values. Every once in a while, this will cause a
garbage collection. If the OCaml value for point has already been
calculated and saved aside (i.e., if the parameters to caml_callback2
are evaluated left to right), this can cause the calculated value to
become invalid before the call happens. This will lead to trouble:
either a crash (as here) or, worse, the wrong answer.

(If my calculations are correct, the expected fraction is indeed 9/64,
or 0.140625.)

In retrospect the problem is obvious, but I’ve wondered for years
whether this construct is OK. As far as I can tell it isn’t
explicitly forbidden by any of the GC Harmony Rules. In
many ways, though, it’s related to Rule 4: the calculated value to be
passed is like a global value, in that it’s outside the reach of the
CAMLlocal() macros.

A good rule of thumb seems to be that you shouldn’t write an expression
as an argument to a function if it can cause OCaml allocation. If
necessary, evaluate the expression before the call.

I hope this may help some other humble OCaml developer seeking to attune
his or her life with the Garbage Collector. If you have any comments or
encouragement, leave them below or email me at jeffsco@psellos.com.

After successfully installing OCaml and Ocaide plugin for Eclipse on a Mac, I now think it's straightforward, however, before I completely forget the pain involved in figuring out what goes where. I wanted to put down some pointers for noobies like me, in hope that it will save people some valuable time.

In general, ocaml.org is a great spot for finding any information related to OCaml. Installing OCaml is no different, you can read all about it here. There is tons of very useful and detailed i…

After successfully installing OCaml and Ocaide plugin for Eclipse on a Mac, I now think it's straightforward, however, before I completely forget the pain involved in figuring out what goes where. I wanted to put down some pointers for noobies like me, in hope that it will save people some valuable time.

In general, ocaml.org is a great spot for finding any information related to OCaml. Installing OCaml is no different, you can read all about it here. There is tons of very useful and detailed information that I highly recommend anyone who is interested in OCaml to check out. In this post I just summarize the steps it took me to get from zero to OCaml installed (hero!).

Feel free to correct me or share your own personal successes and experiences.

Before moving on to the next step, make sure you LOG OUT of your user account so that the X11 install can complete.

Check to see that the directory /usr/X11/lib exists and that it contains the file libX11.6.dylib (this should have been created when you installed XQuartz). If this isn't the case, try reinstalling XQuartz and then contact the course staff for help.

[MM] Removed these instructions, because ~/lib isn't on the default PATH for Macs, and because with most recent SW updates this step isn't necessary.

From the terminal, in your home directory (e.g. /Users/stevez) create a directory called lib (if one does not already exist):

> cd > mkdir lib

Create a symbolic link in that lib directory to the libX11.6.dylib file:

> cd ~/lib > ln -s /opt/X11/lib/libX11.6.dylib libX11.6.dylib

Next, install the Java Development Kit (JDK) 7. You can download it from this this site (after accepting the terms & conditions). Make sure you do this step before installing Eclipse or OCaIDE.

[Update] As was pointed out by numerous stackoverflow posts like this one , Eclipse IDE tend not to recognise environment variables. The solution is to wrap the execution into a bash script. It works, however, I couldn't manage to make it work with the MacOS launch database. Even after the update of the Info.plist, Eclipse still won't recognise environment variables. At this stage, I run the script manually and it works.

\t tells you the type of the thing under the cursor (if you compiled with -annot)

Thomas also adds

"Also, Vim can then parse the output of the compiler and jump to the correct location."

However, he doesn't say how to do that. I searched the web and eventually figured it out. So here is what you need:

Makefile to compile you code

Vim 7.4+ (earlier versions don't seem to support it.

and the knowledge how to use vim's quickfix

Let's tackle these one by one:

1. Here is a sample of a Makefile:

NAME =

OCAMLBUILD = ocamlbuild -use-ocamlfind -classic-display

PACKAGES = -package -tag thread

all:

$(OCAMLBUILD)$(PACKAGES) filename.byte

clean:

rm *.byte

test:

$(OCAMLBUILD)$(PACKAGES) filename.byte

2. Make sure you have an updated version of Vim.

3. You can read the full documentation on it here, or check this quick and what I thought I useful, summary provided by this stack overflow answer - run: make and then:

:copen " Open the quickfix window:ccl " Close it:cw " Open it if there are "errors", close it otherwise (some people prefer this):cn " Go to the next error in the window:cnf " Go to the first error in the next file

That's all folks! Enjoy.

Please let me know if above helped you or you whether you encountered other issues so I can update the post accordingly.

1 After some issues with PATH variables in Eclipse, I am seriously considering converting to Vim.