Update Command_rpc.Connection to check the program before exec'ing it.
The filename must now be absolute, exist, and be executable. Previously
errors with nonexistent or nonexecutable files would only be found out
after forking.

Change Command_rpc.Command to use Versioned_rpc.Callee_converts instead of
Versioned_rpc.Both_convert so that commands can be constructed without client-side
conversions. Clients remain free to use conversions or not, as appropriate.

Removed val rpcs from Callee_converts interfaces because nothing appears to use it,
and Both_convert does not provide it. Now Both_convert.S can be supplied to satisfy
Callee_converts.S.

Add simple example of Command_rpc

Add Deferred_cache.

Fixing a couple of issues noticed in Command_rpc:

If propagate_stderr is false, the child's stderr is now drained instead of
ignored.

When connections are closed, stderr is now closed as well, which prevents
a file descriptor leak if the child process is unresponsive.

async_extra

N.B. some changes happening for this release are not listed in this changelog
since they appear only as a consequence of changes in core or async_kernel.

When Transfer.Writer.send* raises, send an error to the client.

Add a new rpc that enables a "push" rather than a "poll" model.

Switched to PPX.

For connected UDP sockets, expose send in the same fashion as sendto.

Tcp.Server is documented to refuse excess connections beyond
max_connections + max_pending_connections, but it treats them as
pending connections in our standard OS configuration. In fact,
research indicates that the documented behavior is nearly impossible
to obtain directly and consistently from listen.

Clarify the name and role of the backlog argument to listen and
rename and update documentation for max_pending_connections to
clarify what it actually does, in light of some research:

`listen` does not generally respect the backlog argument as an
upper limit, but as a lower limit (mod `tcp_max_syn_backlog`) and,
with `tcp_abort_on_overflow=0`, `listen` will ignore excess
connections rather than actively refusing them.
(With `syncookies=1`, this can look like an indefinite backlog.)

Existing, working code can substitute max_pending_connections ->
backlog and move on. The behavior is not changed.

When possible, consider architecting applications so the server can
simply accept and close excess connections, rather than relying on the
listen backlog to return an active indication to the client that
they won't be serviced. To make sure the client receives an RST
rather than an orderly shutdown, you can set the linger time to 0
before closing the socket. (Added to unit tests.)

Direct Tcp.Server support for this paradigm is left for future work.

Make Rpc_low_latency_transport treat disconnections as eof, like
Async_unix.Reader does.

Add an implementation of Mvars to Async

Allow custom handling of missed async_rpc heartbeats.

adds a configuration limit on the number of tokens that can be in-flight

Replace an #include <sys/errno.h> by #include <errno.h>.

Fixes janestreet/async_extra#4

Added Tcp.Server.sexp_of_t

Adds Rpc.Pipe_rpc.dispatch_iter, plus a bunch of additional types to support
it. The main reason for this is to reduce space usage: Pipe_rpc.dispatch
followed by Pipe.iter_without_pushback consumes ~105 words in the steady state
(i.e., when no messages are coming in) while dispatch_iter consumes ~15. I'm
sure dispatch can be improved a lot, but a pipe by itself is 46 words, so it
can't possibly become as small as dispatch_iter.

Both cases can be made smaller by making Connection.response_handler a GADT
instead of a closure. I plan to do this later.

One annoying property of the interface is that the only way to cancel
a subscription is to use Pipe_rpc.abort, which has a terrible interface.
The logical way to improve the interface is to return a record of
a Pipe_rpc.t, a Connection.t, and a Query_id.t, which allocates an
additional few words. I'd kind of like to do this but it seems counter to the
goal of reducing space usage.

Added Tcp.Server.listening_on_address, so that one can get the
address a server is listening on, as compared with listening_on,
which just returns the port.

Marked Command.async_basic as deprecated using the appropriate ocaml attribute.

Extend the interface of Persistent_rpc_client to make the "address"
type - previously fixed as Host_and_port.t - abstract. This is
helpful for integrating with libraries that have a different notion of
an address, e.g. rpc_discovery_lib.

Typed_tcp mutated a Hashtbl while iterating over it when closing.

Added Async.Bus.first_exn, which takes a bus and a function, and
returns a deferred that becomes determined when the first event is
published to the bus for which the function returns Some.

This function is useful to reduce boilerplate for dealing with
unsubscription.

Reduced the number of threads required by tests in:

async_extra/src/tcp.ml

Added to the error message Bus.subscribe_exn called after first write
the source-code position of the caller, in case there isn't a backtrace,
to make the source of the problem clearer, and to avoid confusion with
other source-code positions of subscribers already in the bus.

Added to Bus.first_exn a Source_code_position.t argument, so that
in the event of subscription failure, we can see who caused the
subscription to the bus.

Added to Tcp.Server.close an optional argument:

?close_existing_connections : bool

This closes the sockets of all existing connections.

Annotate errors returned by the async-rpc library with the name of the RPC for
which the error was returned (if it's an rpc-level error) and a description of
the remote side of the connection (the ip:host if connected via a network
socket).

Improved Async.Udp.bind's error message when it fails to
mcast_join a multicast group.

Change ~callback to ~f throughout the Bus interface

async_find

Initial release.

async_kernel

N.B. some interface change in Core (notably to Hashtbl and Map) implied some
interface change in this package as well, although they are not mentionned in
this changelog.

Switched to ppx.

Improved the Async scheduler's to allocate a handle_fired function
once, rather than every time it calls advance_clock.

The behavior of exceptions raised to a monitor after it returns is
unchanged -- i.e. they are logged, as they have been since 112.28.

Changed Monitor.try_with's ?rest argument from:

?rest : Ignore | Raise
?rest : Log | Raise

This naming reflects the fact that subsequent exceptions are logged,
not ignored.

In Async_kernel, moved Scheduler.set_execution_context from the
Types.Scheduler module to its own file. Because Types is a
module rec, set_execution_context hadn't been inlined and was
called via caml_apply2. In its own file, it will be inlined.

This release creates a new scheduler0.ml, and moves the old
scheduler0.ml to scheduler1.ml.

Fixed a space leak in Pipe due to a pipe holding pointers to its
upstream_flusheds after they are closed. The leak shows up in
Pipe.transfer and related functions, e.g. with:

Pipe.transfer temporary_pipe long_lived_pipe

called repeatedly, in which long_lived_pipe would accumulate a large
number of upstream_flusheds.

The fix is to maintain upstream_flusheds as a Bag.t, and to remove
an upstream pipe when it is closed.

Implement Pipe.of_sequence

Improved the error message when an exception is raised to a
Monitor.try_with that has returned before Async has initialized
Monitor.try_with_log_exn.

Improved the implementation of Monitor.get_next_error, by replacing
the monitor's list of handlers:

; mutable handlers_for_next_error : (exn -> unit) list

with a single ivar:

; mutable next_error : exn Ivar.t

I think this wasn't done originally because of a dependency cycle.
But now that we have types.ml, we can do the clear thing.

Improved the implementation of Monitor exception handling,
i.e. detach_and_iter_errorsto make it clear that Monitor.send_exn
does not run user code -- it only schedules jobs.

This also obviates read_at_most and read_now_at_most, which we
will deprecate in a later release.

Removed a couple helper types, iter and fold, that had been used
to express commonality among functions, but were becoming unwieldy due
to differences.

Changed Pipe's default max_queue_length from Int.max_value to
100.

Added to Pipe.iter_without_pushback an optional argument:

?max_iterations_per_job : int (** default is Int.max_value *)

iter_without_pushback will not make more than
max_iterations_per_job calls to f in a single Async_job; this can
be used to increase Async-scheduling fairness.

Added Pipe.write_if_open which does exactly what it says. This is a
common pattern. Also added a pushback-oblivious variant
write_without_pushback_if_open.

Call sites for these two new functions were introduced wherever I
found that doing so would not introduce any side effects (even
counting allocation) in the case of a closed pipe.

async_parallel

Switched to ppx.

async_rpc_kernel

When Transfer.Writer.send* raises, send an error to the client.

Added Pipe_rpc in Versioned_rpc.Both_convert.

Switched to ppx.

Expose the lower-level registration hook in Versioned_rpc.

Allow custom handling of missed async_rpc heartbeats.

Client-side State_rpc dispatch function does not behave well when the reader side
of the pipe is closed.

It should gracefully abort the rpc instead of raising exceptions, or whatever it currently does.

Add Rpc.Expert.implement and Rpc.Expert.implement' with a similar
interface as One_way.Expert.implement but for 2-way rpcs.

Exceptions raised by an expert implementations are handled as follow:

if the query has already been answered, stop the server (as for one-way expert)

if not, send a Rpc_error.Uncaught_exn (as for 2-way rpc)

Adds Rpc.Pipe_rpc.dispatch_iter, plus a bunch of additional types to support
it. The main reason for this is to reduce space usage: Pipe_rpc.dispatch
followed by Pipe.iter_without_pushback consumes ~105 words in the steady state
(i.e., when no messages are coming in) while dispatch_iter consumes ~15. I'm
sure dispatch can be improved a lot, but a pipe by itself is 46 words, so it
can't possibly become as small as dispatch_iter.

Both cases can be made smaller by making Connection.response_handler a GADT
instead of a closure. I plan to do this later.

One annoying property of the interface is that the only way to cancel
a subscription is to use Pipe_rpc.abort, which has a terrible interface.
The logical way to improve the interface is to return a record of
a Pipe_rpc.t, a Connection.t, and a Query_id.t, which allocates an
additional few words. I'd kind of like to do this but it seems counter to the
goal of reducing space usage.

Adds Rpc.Pipe_rpc.implement_direct, which uses a "direct stream writer" to
write results to the other side, rather than using a Pipe.Writer.t. This
consumes much less memory, ~15 words per open query as opposed to ~225 for
a regular Pipe_rpc implementation.

A large part of the diff in this feature is the addition of a module
Implementation_types, which just contains copies of type definitions from
Implementation and Implementations. This is necessary to handle some cyclic
type definitions (both of these modules now depend on types defined in the other
module).

This is the implementation-side dual of Pipe_rpc.dispatch_iter.

Change Command_rpc.Command to use Versioned_rpc.Callee_converts instead of
Versioned_rpc.Both_convert so that commands can be constructed without client-side
conversions. Clients remain free to use conversions or not, as appropriate.

Removed val rpcs from Callee_converts interfaces because nothing appears to use it,
and Both_convert does not provide it. Now Both_convert.S can be supplied to satisfy
Callee_converts.S.

Annotate errors returned by the async-rpc library with the name of the RPC for
which the error was returned (if it's an rpc-level error) and a description of
the remote side of the connection (the ip:host if connected via a network
socket).

Bring back val rpcs in versioned_rpc modules.

async_shell

Initial release.

async_smtp

Switched to PPX.

Follow Core & Async evolution.

async_ssl

Switched to ppx.

async_unix

In the periodic check for a Writer buffer have too old data,
eliminated allocation and generally improved performance.

This eliminated a large source of allocation in a simple TCP pingpong
benchmark bench/pingpong.

Removed allocation in the Async scheduler's code that detects the
thread-pool being stuck. This involved switching it to use Time_ns
rather than Time.

This eliminates a relatively large source of allocation in a simple
TCP-pingpong benchmark bench/pingpong.

Switched to ppx.

Improved the Async scheduler's to allocate a handle_fired function
once, rather than every time it calls advance_clock.

Added Fd_by_descr.find_exn, to avoid the option allocated by
Fd_by_descr.find. Used it to reduce allocation in the Async
scheduler.

Improved Reader.load_sexp* functions to behave better when loading
from a non files, e.g. a pipe. Previously, it produced an empty error
message because it mistakenly attempted to read the sexp a second time
in order to determine the error position. Now it produces a good
error message, but without the (impossible to obtain) error position.

In Async_unix.Syscall, added a zero-allocation syscall interface,
removing sources of allocation as observed when running a simple TCP
pingpong benchmark (found in bench/pingpong).

Added

val time_spent_waiting_for_io : unit -> Time_ns.Span.t

to Scheduler which returns the amount of time that the Async scheduler has
spent in calls to epoll_wait (or select) since the start of the program.

Switching create to return a deferred allows it to, when there are
no available threads, wait until one becomes available. This, in
turn, avoids rare nondeterminstic failures in programs that make heavy
use of the thread pool and create a helper thread, when the creation
happens at just the wrong time, when no thread is available.

Split out Thread_safe_ivar from the internals of Thread_pool, so
that it can be used in other tests, and in particular in a new
unified test added by this feature.

Make Unix_syscalls.Stats bin-io-able.

Fixed a bug in Thread_safe.block_on_async*, in which the execution
context wasn't properly restored before returning.

Added Reader.read_one_iobuf_at_a_time, which is like
read_one_chunk_at_a_time, except that the user-supplied
handle_chunk function receives its data in an Iobuf.t, and uses
the Iobuf position to communicate how much data was consumed.

This facilitates using reader in scenarios (such as with protogen)
where Iobufs are expected (and presently allocated around the
bigstring at each call) and the calculation of consumed bytes from the
Iobuf is duplicated in few places.

Log.message used to always logs the message, even if its log level was too low.
This has been fixed.

Add writer functions to schedule an iobuf to be written out.

Add Unix.Inet_addr.Stable.

Alter Async.Std.Socket.Address.Inet.Blocking_sexp to expose the
polymorphic variant functions, so that you can include it in a
separate polymorphic variant type.

Also, expose Async.Std.Socket.Address.Inet.__t_of_sexp__ to give a
deprecation message, instead of a message about the function not
existing.

Fixed a bug in Async's In_thread.Helper_thread, which wasn't
finalizing helper threads, due to a bug in Thread_pool, which wasn't
finalizing helper threads. The fix was to move the finalization out
of Thread_pool, where we don't care about it, to
In_thread.Helper_thread, where we do.

Added Scheduler.max_num_threads : unit -> int.

Make Epoll_file_descr_watcher trigger callbacks for error conditions such as closed pipes.

Testing

Three new unit tests, all validating appropriate behavior in the case that a Unix pipe is
opened, then the read end is closed after reading only part of the data sent by the write
end.

A test in writer.ml verifying that Writer.consumer_left is triggered. Before the fix
to Epoll_file_descr_watcher, Writer.consumer_left would never become determined in
this case.

A test in fd.ml verifying that Fd.ready_to is triggered for the writing fd. Before
the fix to Epoll_file_descr_watcher, Fd.ready_to would never become determined in
this case.

A test in linux_ext.ml verifying that Epoll.wait_timeout_after is triggered. This
test shows that epoll reports the ERR flag for the file descriptor in this case, and
therefore that Epoll_file_descr_watcher needs to pay attention to the ERR flag.

Added to Writer.write_sexp an optional ?terminate_with argument,
that specifies how to terminate the string representation of the sexp.
This also makes it clear that the default behavior,
~terminate_with:Space_if_needed, might append a space to the sexp
you are outputting if its representation is not enclosed in either ()
or "" . Sexp.output_hum an Sexp.output_mach do not have this
behavior, so porting non-async code to async could introduce
unexpected differences in the output.

Add an Async wrapper for Core.Std.Unix.getifaddrs.

bignum

Switched to PPX.

The library used polymorphic compare, rather than Zarith.Q's compare, in a
few locations. Fixed this.

Previously stable types in Bignum were defined with unstable types in the scope.
Fixd this.

Update to zarith-1.4

Bignum.of_string needs to handle different formats for its input. The
previous version of the code was trying to parse the common format
(floats), and in case of failure, was attempting to use a different
format (based on the error). This resulted in the string being parsed
twice in some cases.

This version is a complete rewriting of of_string to do the parsing
in one step. The new code for to_string encode an automaton and
remembers the positions of the various elements of the string
(depending on the format).

This feature uses a function which has been upstreamed in the new
version of ZArith (1.4) which is a variant of the Zarith.of_string
function to work with substrings. This variant alone is responsible
for a big part of the performance improvement.

Summary of benchmarks

The new version of the code performs better than the original one in
all cases. The performance improvement are variable depending on the
micro benchmark. See below.

Follow ups

We also tried to implement the lexing engine using OCamllex.
This makes for a much more concise description, but the
performance are significantly lower. OCamllex produces code which
allocates some table and some state, which is avoided in the hand
written code. Also, it will allocate the sub strings matched.

Benchmark results

New version (patch for ZArith + of_substring, reimplementation of of_string)

bin_prot

Bin_prot can be configured to use the primitives to read/write
integers from bigarrays. This was never enabled due to missing tests
that selecting this code path doesn't change the format.

This version add these tests and enable the use of the fast
primitives.

Add benchmarks for all exposed bin_prot read/write functions.
These are intended to check performance regressions.

Remove most use of cpp in bin_prot.

Replace the pre-processor conditionals by runtime one. This make the
code a little less obfuscated.

Remove big literals so that the compiler does not complain in 32bit

core

N.B. Some interface change were made which are not listed here, as they are only
cascading from core_kernel. Look at core_kernel's CHANGES.md file to get a
complete history of the changes made for this release.

Add Core.Command.Arg_type.Export.time_zone

Fix Command.shape to run external programs only once to get their sexp.

Introduces a new variant of Command.t called Proxy. The Exec variant represents the
top-level command of an external executable that has not yet been run; the Proxy variant
represents an arbitrary subcommand that has been extracted by running an external
executable. Command.exec constructs an Exec variant; Command.shape of an Exec variant
runs the executable and generates a tree of Proxy variants representing all the
information from the generated sexp, so the executable will not need to be re-run.

A version of recvmmsg that pre-allocates and reuses the iovec
record. Profiling indicates this is a non-trivial amount of our I/O
loop (under very heavy load).

Since nobody is using the heavyweight features of the existing
recvmmsg, replace it with the lightweight one. This leads to minor
but important changes to the interfaces of
Iobuf.recvmmsg_assume_fd_nonblocking and Udp.recvmmsg_loop.

Switch to ppx.

Time.set_sexp_zone affects t_of_sexp

If we're willing to read sexps without zones, we should be willing to let you
control what timezone they're read with.

Fixed Iobuf_packet.iter, which behaved incorrectly if the packet
didn't start at the lo_min of the iobuf. It used Iobuf.rewind when
it should have used Iobuf.Lo_bound.restore.

Added @@deriving compare to Iobuf.Bound.

Add ability to Time.format for a specific time zone

Make more information available via Command.Shape.t

Expose machine-readable info on anonymous arguments.

Remove unnecessary rebinding of (^/) in core_filename.ml. It had one
call site and wasn't exposed. The (^/) everyone uses comes from
std.ml

Add getifaddrs to Core.Std.Unix.

Handles Packet (for interfaces that do not have an address on Linux systems only), IPv4 and IPv6
address families.

Implement Time_ns.Span.to_string_hum by analogy to Time.Span.to_string_hum.
The code and tests are essentially copied from "lib/core/src/span.ml".

Remove our stubs for Unix.stat.

They were upstreamed in 4.02.2.

core_bench

Switched to ppx.

core_extended

N.B. Some interface changes occured in Core which are repercuted in this
package, they are not all list in this file though.

Switched to PPX.

Upgrade Interval_map.t with monad operations.

Update the interval_map_intf.ml file to try to make the documentation
clearer.

This mostly constitutes splitting out the core operations into a separate
module type earlier in the file, so that their documentation occurs before
the various more specific module types in reading order.

Various bits of documentation have been tweaked with examples or laws.

Add underscores to color print names
Improve and uniformize the behavior of colorprintf functions at the cost of changing the type slightly

Added a function for sequencing computations stored in a total map:
Total_map.sequence.

Added Core.Bus, a publisher/subscriber system within the memory
space of the program. This is a synchronous version of Async.Bus.

Added Core_map.fold2 (fold based on the contents of two maps side-by-side).

Core.Interfaces defines the Unit module type to be sig end.
Increase uniformity with its other definitions by defining it to be
Unit.S instead.

Adapt Core_random.int to accept larger values than 1 lsl 30.

Mark the Sexpable.Of_* and Binable.Of_* functors as stable.

In Core_char.int_is_ok, used by of_int and of_int_exn, use int compare
instead of polymorphic compare.

Fix a few files where toplevel side effects might not be running
when we don't pack libraries anymore and use -no-alias-deps.

In Char.For_quickcheck, memoize construction of the filtered chars generators, since if
they are used once, they are likely to be used many times, and the construction is costly
compared to generating a single char.

Made Random.self_init by default raise if used in inline tests.
One can opt out by passing ~allow_in_tests:true.

In core_hashtbl.ml, maybe_resize_table allocates the same closure in each iteration of a
for loop. Allocate it just once.

Hashtbl.remove_one and Hashtbl.remove_multi are the same function, written twice.
Remove remove_one and replace uses with remove_multi.

Bigstring.unsafe_{get,set}-{,u}int8 used the generic bigarray
access function without a type annotation. As a result the compiler
generated a call to the generic C function.

Fixed this by adding type annotations.

Add new functions to map that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules.

Made Unpack_buffer.Unpack_one monadic so that users can easily
compose file parsers

Added a couple simple parsers as examples and for testing.

Avoid use of polymorphic compare in Quickcheck.
Make Quickcheck.Generator.bind_choice lazy: do not eagerly descend into all branches.

Reduces memory overhead by setting a threshold on the probability of choices that are
remembered and discarded by Quickcheck.iter and friends.

Motivation: Currently, Quickcheck.iter and related functions guarantee never to repeat
a choice from a generator. This winds up recording every choice ever made, which for a
lot of generators is a prohibitive cost in space, and most of the recorded values are very
unlikely to be repeated anyway.

Implementation: This feature sets a probability threshold below which choices will not be
remembered. Choosing a fairly low, but still non-zero, threshold means values are still
very unlikely to be repeated, but memory usage stays low.

As of this version, the benefits of "forgetting" unlikely-to-repeat values:

In Core_hashtbl, the add_worker function used a bool ref both internally and to pass
to Avltree to track whether a new key is added. This was allocated on every
call to add or set, and set didn't even use its contents.

This version pre-allocates the bool ref inside each Core_hashtbl.t and reuses it. It
still can't be a mutable field because it does need to be passed to Avltree.

In Core_hashtbl.add_worker, removed a match that avoided calling Avltree.add, but
actually did hurt performance overall.

Perhaps at some point before cross-module inlining, this was a helpful optimization.
Right now it bypasses the mutation inside Avltree, so replacing a value in a
non-colliding bucket (a Leaf) causes unnecessary re-allocation of the leaf.

Make Hashtbl functions raise an exception if a callback passed in as an argument mutates
one of the hash tables being worked on.

Usually, though not always, this comes up for iteration functions. Once a hash table has
been mutated, it is unsafe to continue operating on it, as its structure may have changed.
Buckets and their contents may have been moved or resized; continuing may result in
skipping key/value pairs, repeating key/value pairs, or executing unsafe operations.

This feature adds a mutation_allowed flag to hash tables. Each mutating operation first
checks the flag, and raises if it is not set. Each operation with callbacks that must not
mutate unsets the flag before calling the callbacks, and restores the flag's original
value when it finishes.

We compared the timing of this implementation to an alternate implementation using a
mutation counter, and the time and space used for this implementation was much better for
iteration and within epsilon of the other for single-key operations like set.

Array function names related to zipping are all over the place. Make
them match List, which has a nice uniform naming scheme.

Rename combine -> zip_exn

Rename split -> unzip

(zip remains named as zip)

Add ~key and ~data labels to Hashtbl.filteri_inplace

Added Hash_set.to_hashtbl, by analogy to Set.to_map.

Since we are mutating avltrees in place, make sure the compiler sees
the type parameters as invariant.

Tested that a segfaulting example doesn't compile anymore.

Add label f to Hashtbl.change, Map.change, & family.

Introduce the new function update in those modules, which enforces
statically the presence of a resulting value

The motivation for the introduction of update is that in an overwhelming
majority of the places where Hashtbl.change is used in our codebase, it is
statically known that a new value shall be computed and stored. The use of the
dynamism offered by change, which can return an option, is error prone.

The addition of the label is considered acceptable in consideration to external
libraries depending on core, because a missing label is just a warning, and we
do not guarantee stability in the presence of -warn-error = true.

Changed Source_code_position.t from:

@@deriving bin_io, sexp

to:

@@deriving sexp_of

and made sexp_of use the human-readable format, "FILE:LINE:COL",
rather than the unreadable format. Removed
Source_code_position.t_hum, which is now obsolete.

If one wants a serialized source-code position, one can use
Source_code_position.Stable.

Added Ref.set_temporarily, for temporarily setting a ref to a value for
the duration of a thunk.

val set_temporarily : 'a t -> 'a -> f:(unit -> 'b) -> 'b

Add the function singleton : 'a -> 'a t in the stack containers. It cannot be
added to Container.S directly because some container never have exactly 1
element.

Made Core.Array match Invariant.S1.

Change the interface of Make_iterable_binable* to give the control
back to the user when deserializing Bin_protted data.

Improve the bin_prot deserialization of Maps and Sets.
We construct a balanced tree directly instead of relying on Map.add / Set.add.
This is possibile because the size of the map is known and elements are sorted.

The complexity goes down from n.log(n) to n.

In case the comparison function changes (and the invariant is not respected),
there is a fallback to reconstruct the whole map from scratch.

Add a function to blit a Rope.t into a Buffer.t.

Hashtbl differs from some other core containers with idiosyncratic naming of iteration functions. Change to be consistent and to more closely match the conventions for List and Array.

Hashtbl:

Copy iter -> iteri.

Add a deprecation tag to iter.

Made Bag.invariant and Doubly_linked.invariant match Invariant.S1.

Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current Map.iter vs Map.map vs Map.mapi). Change to be consistent and to more closely match the conventions for List and Array.

Map:

Copy filter -> filteri.

Add a deprecation tag to filter.

Map differs from some other core containers with idiosyncratic naming
of iteration functions. The current Map name conventions are also
internally inconsistent as well (ex: current Map.iter vs Map.map
vs Map.mapi). Change to be consistent and to more closely match the
conventions for List and Array.

Add Ordering.to_int which can be useful when one is writing a comparison
function. Instead of dealing with the int directly, one can return Ordering.t
values and transform them later into ints.

Float.int_pow: Fast computation of x ** n when n is an integer.

Make Core_kernel.Std.Nothing.t enumerable. There's no particular reason
not to.

Minor improvements to queue interface

Call Caml.Pervasives.do_at_exit before printing an exception and exiting

The default ocaml uncaught exception handler does this. It is
especially useful for curses applications as the at_exit handler has
a chance to put back the terminal in a good state before printing the
exception and backtrace.

Do the same in Core and Async.

Removed big literals so that the compiler does not complain in 32bit

Add List.range', a generalization of List.range.

Add some functions to Map that are present in Hashtbl:

remove_multi

partition_tf

partitioni_tf

partition_map

partition_mapi

Add a Map.nth_exn as a missing complementary function to nth

Renamed Validate.fail_sexp as fail_s, to follow our new *_s
convention for Sexp.t-taking functions.

Sequence.split_n_eagerly returns a pair of sequences, but every element of the
first sequence has already been evaluated by the time it returns. This feature
just makes the first component of the tuple a list instead of a sequence, and
renames split_n_eagerly to split_n.

Additionally, this feature adds a new chunks_exn function, which just applies
split_n until the input sequence is empty.

Removed Timing_wheel's default alarm_precision, to force people to
think about the precision they want when they create a timing wheel.

In Timing_wheel.Config.sexp_of_t, used @sexp_drop_default with
level_bits.

Write a better-performing Array.filter_mapi function, and implement
Array.filter_map, Array.filter_opt, Array.partitioni_tf, and
Array.partition_tf in terms of it.

Slightly worse for zero-length input arrays, about unch'd if we're filtering out
almost everything (eq_zero), better on most everything else.

Install the sexp exception printer sooner so that we can get proper
%test_result ... errors in things that come before core_kernel.

In Stable_unit_test.Make functors, include all test failures rather
than just the first. This is useful for updating batches of expected
bin_io results when stabilizing a module.

Remove an unnecessary cast in or_error.ml

core_profiler

Switched to ppx.

Minor adjustments to the command line of profiler_tool.exe:

Make '-%' an alias for '-percentile'

Make '-percentile' accept a comma-separated list of numbers

Add '-median' argument that is equivalent to '-percentile 50'

email_message

Bugfixes and minor API improvements.

incremental

Add README.org to Incremental.

Added some type annotations based on comments by @def-lkb about lack of
principality.

Switched to ppx.

jenga

Restructure the code in a way that allows to build binaries that statically link jenga
with the rules.
This is useful because some debugging/profiling tools don't work in the presence of
dynamically loaded code very well.

Switch to PPX.

Change the gc info output by jenga so it shows heap size and top heap size, instead of
live and heap size. The live part is not super useful given how random it is. I have seen
cases where jenga was using 20GB during building and jenga reported a heap size of 13GB at
the end so the top heap size avoids being tricked.

First half of the fixes no packing: sharing the structures of dependencies, so they take
less space on disk (and in memory as well, when they are loaded from disk, but not really
when building from scratch given the way we will use them).

The sexp format also has sharing, because it would also blow up in size otherwise (this is
different from the interning of paths, where the interning saves a constant factor). And
of course, it makes it possible to see the actual on-disk representation which is nice.

Also fix unhelpful error (contains no information) when the db can't be loaded.

Break the thing that avoids rerunning rules when the set of dependencies decreases. I
think it was never useful anyway.

Better error on duplicate targets in the same rule.

To prevent running more than one jenga in a repository, use a local
lock rather than an nfs one. We need a transition period though, so
for now we use both kinds of locks.
Building on nfs is slow, so I don't think there's any downside is not
supporting nfs this way. And maybe inotify doesn't work. The upside is
that we don't step into Lock.Nfs bugs where if a process is
interrupted at the wrong time (when the two lock files are empty) the
locks won't be cleaned up automatically, forcing someone to get rid of
the lock files manually.

Also rename .jenga/.jenga.* to .jenga/*, because all these prefixes
are annoying.

Added a couple of options to turn off some part of jenga, which I used
to check how they impacted performance, and could still be handy
later.

Optionally display additional information about much allocation was done, at the end of
builds. Used it to try to improve memory usage of full tree builds without actually
doing full tree builds.

Make stat'ing faster.
Hash cons some tenacious that build mtimes map to avoid a huge
increase of memory use.

Got rid of noise when stopping jenga.

Some changes to the implementation of Tenacious to improve memory efficiency.

Includes the following changes:

Remove strong_refs field of Heart.fragile type and instead insert links from
the clientsRing.t to its parent using Ring.keep_alive.

Replace the Ring.t in the triggers field of Hearts.fragile with an
Ivar.t since all uses of triggers were producing their own equivalent
IVar.ts. Remove the functions broken by this because they were unused.

Make the Tenacious.t type a concrete datatype, and optimize pure
computations by partially evaluating this datatype directly in the pure case.

Rather than building separate Heart.ts for cancellation and the result,
split the cancellation heart into two hearts cancel and dep and then use
dep as the result heart. This means that a tenacious is cancelled if either
cancel or dep is broken, and it must return a heart representing the
validity of its result combined with dep.

Cursory benchmarking indicates a 23% improvement in maximum resident set size and a 10% improvment in (user) execution
time when building the lib directory from scratch.

Adding some sexp_of functions, since they're always missing and
it's a pain when debugging.

Adding direct support for Dep.map.
Even now that Tenacious is smarter, this still creates less binds. Doesn't seem to make
much of a difference (perhaps 3-5% less allocation, on a null build of lib), but if
nothing else, it's much less surprising to think that Dep.Map becomes Tenacious.Map.

Added Dep.List.concat.

Make it possible to turn off the behavior where jenga rejects commands that output on
stderr.

It increases slightly the footprint of the in memory db, but the difference is tiny
compared to the rest of the memory usage.

Added a few tests about Jenga_lib.Api.Reflect

ocaml_plugin

Switch to ppx.

Allow ppx-style code to be loaded by plugin-applications build using ocaml_plugin.

Follow Core & Async evolution.

patdiff

patdiff -location-style omake should print the line number of the
first difference in each hunk, skipping context lines.

Switched to PPX.

Added binding in patdiff to use the newly minted colors of Ansi_terminal. This
will be used notably by patdiff4 to produce better ddiff.

Also, have the module Color and Style implement and export Comparable.S.
This is useful for example to dedup styles from a list of styles without relying
on the polymorphic equality.

Make it so that if you pass -warn-if-no-trailing-newline-in-both false
then you get the warning only when one file has a trailing newline and
the other file does not.

If you pass -warn-if-no-trailing-newline-in-both true or omit this
flag, then you get the current behavior of warning for each file
independently.

Patdiff's unified-tests currently render colors codes in angle
brackets. Change them to square brackets. Square brackets are word
boundaries, so we'll get more legible diffs when tests fail.

Simple code change in patdiff to prepare more changes in patdiff4. This change
is a pure refactoring and has zero runtime change. Just moving some functions
around.

patdiff_core.ml is a very long module. start extracting module from it. start
with format. in the process, expose in a private fashion the record Rule.t.

Continue on splitting the file patdiff_core.ml into smaller pieces.
In this version, we extract each output mode into its own file.

Kill the generation of html diffs in patdiff. There are good third party tools
that can convert efficiently ansi texts to html directly.
We plan on simplifying a bit the patdiff source code to increase its
maintainability, and dropping the requirement of producing html output seems a
step in the right direction.

patience_diff

ppx_assert

ppx_bench

So that a @@deriving inside a let%test dropped by
ppx_inline_test_drop doesn't cause a failure.

ppx_bin_prot

Minor changes, nothing worth mentionning.

ppx_compare

Follow evolution of Ppx_core and Type_conv.

ppx_core

Kill the nonrec rewrite done by typerep. It is no longer needed since
4.02.2, we kept it only for compatibility with the camlp4 code.

Merlin uses @merlin.* ... attributes in different places. Which ppx_driver
reports as unused.

Introduce the concept of reserved namespaces.
When one declares the namespace "foo" as reserved then:

foo.* will never get reported as unused

it is impossible to Attribute.declare "foo.*"

Mark the "merlin" namespace as reserved by default.

Don't print:

Extension `foo' was not translated.
Hint: Did you mean foo?

OCaml makes no distinctions between "foo" and
{whatever|foo|whatever}. The delimiter choice is simply left to the
user.

Do the same in our ppx rewriters: i.e. wherever we accept "foo", also
accept {whatever|foo|whatever}.

Avoid stupid hints like this one:

Attribute default' was not used.
Hint:default' is available for label declarations but is used here
in the context of a label declaration. Did you put it at the wrong
level?

Update the API for the common case of extension point expanders.

Make it simpler to define ppx rewriters that locally expand extension
points, which is the majority of our non-type-conv rewriters.

Such expanders are run inside the same Ast_traverse.map in a
top-down manner which:

probably improve speed

help with rewriters that capture a pretty-print of their payload

help with rewriter that interpret some extension points in a special
way inside their payload

Fix the order in which errors are reported by ppx rewriters.
Make them be reported in the same order as they appear.

Mark attributes as handled inside explicitly dropped pieces of code.

So that a @@deriving inside a let%test dropped by
ppx_inline_test_drop doesn't cause a failure.

ppx_csv_conv

ppx

An umbrella feature for development on ppx syntax extensions.
All work except for rebasing should be done in subfeatures.

Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml

ppx/changes-for-public-release

Changes required for the first public release of our ppx rewriters

Improved the type_conv export to ppx_deriving

improved how it works, to enable it we just need:

Type_conv.Ppx_deriving_exporter.set (module Ppx_deriving)

Type_conv.add_alias now takes as argument registered type_conv
deriviers. This ensure that we can resolve aliases right from the
registration time. This simplify the export to ppx_deriving

Split some ppx rewirters

Split some functions out of their main ppx_XXX library that does the
registration with ppx_driver or ppx_type_conv.

For instance:

ppx_sexp_conv defines sexp_of_quote that is used by ppx_assert

ppx_here defines ast_of_pos that is used by ppx_fail and ppx_assert

Making ppx_assert depends on ppx_sexp_conv and ppx_here is not good as
it enables @@deriving sexp and %sexp_of: ty even if the user only
writes ppx_assert in the jbuild.
This is especially problematic for the public release as it makes
ppx_assert incompatible with ppx_deriving.

This feature moves these functions into libraries called
ppx_XXX_expander (and ppx_inline_test_libname for ppx_inline_test).

Move Ppx_type_conv.Std.Type_conv_path to Ppx_core

It's not type-conv specific anymore and is needed by libraries that
don't use type_conv otherwise.

Rename some old runtime libraries

pa_test_lib --> ppx_assert_lib
pa_bench_lib --> ppx_bench_lib

Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml

ppx

An umbrella feature for development on ppx syntax extensions.
All work except for rebasing should be done in subfeatures.

Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml

ppx/delete-make-at-the-end

Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end, which was a hack
to remove warnings. We can do it better now, and because this is only for signatures, the
code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't
apply.

For users

This feature moves a few values up in signatures. For instance in this
interface sexp_of_t is now the first value instead of the second one
as before this feature:

type t `@@deriving sexp_of`
val x : int

In some cases this caused the OCaml compiler to complain about items
in the signature being re-ordered. In these cases the signature was
adapted to match the expected ordering.

Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml

ppx

An umbrella feature for development on ppx syntax extensions.
All work except for rebasing should be done in subfeatures.

Testing

ppx_custom_printf

OCaml makes no distinctions between "foo" and
{whatever|foo|whatever}. The delimiter choice is simply left to the
user.

Do the same in our ppx rewriters: i.e. wherever we accept "foo", also
accept {whatever|foo|whatever}.

Fix missing location in errors for broken custom printf example like:

printf !"%{sexp: int" 3;;

Update to follow Ppx_core evolution.

ppx_driver

Disable safety check when code transformations are used as standard
"-ppx" rewriters.

Introduce reserved namespaces, see Ppx_core's changelog.

Pass errors as attribute with -dparsetree to avoid
"Error while running external preprocessor".

Update to follow Ppx_core evolution.

ppx_enumerate

Update to follow type_conv evolution.

ppx_expect

Initial release.

ppx_fail

Added a README.md

Update to follow Ppx_core evolution.

ppx_fields_conv

The iter function generated by ppx_variants_conv and ppx_fields_conv allowed
one to give function which returned values of arbitrary types as iter function.
This release constraint these functions to return unit.

N.B. the signature generated by the use of @@deriving variants (resp. fields)
in interface already constrained the type to unit.

Update to follow type_conv's evolution.

Add Fields.make_creator to ppx_fields_conv's readme, since it appears
to not be all that deprecated.

ppx_here

Make ppx_here translate [%here] instead of _here_.

Update to follow Ppx_core evolution.

ppx_inline_test

Support literate-style .ml files that allow ocaml code interleaved with expected output
annotations. Compiling with the ppx_expect_test generates a program that outputs the
original source file, but with the actual output substituted for the expected-output
annotations. Then we can pat-diff the original file against the output file.

Testing

Examples in the test/ and example/ folders.

Expect-tests can now be written inline in libraries by using let%expect_test.

The runtime library has been split into two components: the test runner, which
collects the output of the test body, and registers enough information to
construct the *.ml.corrected file from the input; and the test evaluator,
which compares the test output against the expected output and generates the
output files.

Update to follow Ppx_core evolution.

When an exception is raised inside a let%test_module, display the position
and name of the TEST_MODULE, same as for the let%test.

Mark attributes as handled inside explicitly dropped pieces of code.

So that a @@deriving inside a let%test dropped by
ppx_inline_test_drop doesn't cause a failure.

ppx_optcomp

Change the way optcomp resolve filenames in #import directives

Do the same as cpp, i.e. for relative filenames, consider they are
relative to the directory of the file being parsed. This doesn't
matter internally as build commands are always executed from the
current directory, but it matters for the public release as everything
is executed from the root.

ppx_sexp_conv

Trying to improve the tests in ppx_sexp_conv because they are a mess.
At least all tests are automatic now. And more things are tested like
the sexpification of exceptions.

Before, v_of_sexp would never manage to read B. This problem is
now fixed if you usesexp_polyonuinstead ofsexp, and if you
don't, you get an "unbound value __u_of_sexp__". People should use
sexp_polywhen they have a polymorphic variant type that is not
syntactically a polymorphic variant, but in practice it's simpler to
replacesexpbysexp_poly` when faced with the error above.

The need for sexp_poly should happen only in one new case: an
implementation says type u = t@@deriving sexpbut the interface
says `type u =A@@deriving sexp. (the old case where it was
already needed is when you have an interface that says `type u = t
`@@deriving sexp and in some other implementation you try to say
type t = That_module.t | A@@deriving sexp`).

ppx_type_conv

Kill the nonrec rewrite done by typerep. It is no longer needed since
4.02.2, we kept it only for compatibility with the camlp4 code.

Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end,
which was a hack to remove warnings. We can do it better now, and because this
is only for signatures, the code generation issue what we had in
simplify-type-conv-ignore-unused-warning doesn't apply.

Update to follow Ppx_core evolution.

ppx_typerep_conv

Update following Ppx_core and Type_conv evolution.

Add a README.

ppx_variants_conv

The iter function generated by ppx_variants_conv and ppx_fields_conv allowed
one to give function which returned values of arbitrary types as iter function.
This feature constraint these functions to return unit.

N.B. the signature generated by the use of @@deriving variants (resp. fields)
in interface already constrained the type to unit.

Update to follow Type_conv evolution.

re2

Switched to PPX.

Add Re2.Parser.any_string combinator.

There are no tests because any_string is constructed only from the tested
API and there's almost no interesting properties of it that can be verified.

rpc_parallel

Switched to PPX.

Expose the connection_timeout argument in rpc_parallel. This argument
exists in Rpc_parallel_core.Parallel, but it is not exposed in
Rpc_parallel.Parallel.

Allow custom handling of missed async_rpc heartbeats.

Give a better error message when redirecting output on a remote box to a file
path that does not exist.

remove unncessary chmod 700 call on the remote executable

Give a clear error message for the common mistake of not making the
Parallel.Make_worker() functor application top-level

Make errors/exceptions in Rpc_parallel more observable

Make stderr and stdout redirection mandatory in order to encourage logging stderr

Clean up the use of monitors across Rpc_parallel

Fix bug with exceptions that are sent directly to Monitor.main
(e.g. Async_log does this)

Add the ability to explicitly initialize as a master and use some subcommand for the
worker. This would allow writing programs with complex command structures that don't have
to invoke a bunch of Rpc_parallel logic and start RPC servers for every command.

Add the ability to get log messages from a worker sent back to the master.
In fact, any worker can register for the log messages of any other workers.

sexplib

Switch code in lib subdir to ppx-style.

textutils

Switched to PPX.

Fixed a bug where the computation of cell heights could cause division by zero
in some cases.

Expose the constructors of Ascii_table.Align.t so that we can write

Column.create ~align:Left ...

instead of

Column.create ~align:Align.left

typerep

Add whether record fields are mutable.

typerep_extended

Switched to ppx.

Kill the nonrec rewrite done by typerep. It is no longer needed since
4.02.2, we kept it only for compatibility with the camlp4 code.

113.00.00

async

Added Async.Std.Printf module so that one doesn't unintentionally use
blocking Core.Std.Printf functions in an Async program.

async_inotify

async_kernel

Switched Lazy_deferred to use Or_error.t rather than ('a, exn) Result.t.

Note: There is difference in the run argument between Monitor.try_with
and Monitor.try_with_or_error. In this module, the function is called
already in a closure inside a bind, so that difference is acceptable.

Made Deferred match Invariant.S1.

Improved Async_kernel.Scheduler.run_cycles_until_no_jobs_remain to use
Timing_wheel.fire_past_alarms rather than sleeping.

Added Quickcheck module.

Reworked the initialization of Monitor.try_with_ignored_exn_handling to log
exceptions using Async.Log so that it doesn't rely on top-level effects,
which may not happen without packed libraries.

async_rpc_kernel

Fixed race in Rpc that caused double connection cleanup.

Two errors, Connection_closed and a Writer error,
(Uncaught_exn(monitor.ml.Error_((exn(\"writer error\"....)))))),
occurring at the same time will cleanup the connection twice and call
response_handler of open_queries twice with two different errors.

Added Rpc.Pipe_rpc.close_reason and Rpc.State_rpc.close_reason, which
give the reason why a pipe returned by an RPC was closed.

These functions take the IDs that are returned along with the pipes by
the dispatch functions, so the interface of dispatch did not need to
change.

Made Rpc.Expert.dispatch expose that the connection was closed, just like
One_way.Expert.dispatch.

Expose the name of the Versioned_rpc.Menu RPC.

async_smtp

Improve the async_smtp client interface so that it is suitable as a
replacement for Core_extended.Std.Sendmail.

async_ssl

Added Ssl.Connection.close.

async_unix

Made Async dump core when it reports a "bug in async scheduler".

There is no change for toplevel unhandled user exceptions, for which Async
does not dump core.

Added Dump_core_on_job_delay.dump_core function, which exposes the
core-dumping functionality in the C stubs for Dump_core_on_job_delay.

Made Dump_core_on_job_delay.How_to_dump an ordinary variant and moved it
into Async_kernel.Config.

Changed Thread_safe_pipe functions that write to the pipe to take an
additional argument, an If_closed.t, that says how to behave if the pipe is
closed.

The previous behavior is achieved with ~if_closed:Raise. One can also
now use ~if_closed:Return to they return a variant reporting whether the
pipe was closed, rather than raising.

Returning a variant allows callers to distinguish the pipe-closed case from
other errors. This change also allows us to do a a single acquisition of
the Async lock, with the pipe-closed check synchronously immediately
preceding the operation, avoiding a race.

Added Fd.with_file_descr_deferred_exn.

Improved the performance of Clock.every, and in particular reduced its
allocation.

It now allocates much less, especially with ~continue_on_error:false.

Handled Clock.every's ~stop argument directly using timing-wheel
alarms, rather than using Deferred.choose.

Slightly changed the behavior of

Clock.every' f ~continue_on_error:false

in the corner case where f raises but its result also becomes
determined. Prior to this feature, iteration would stop. After this
feature, iteration will continue, because ~continue_on_error:false
just looks at the deferred resulting from f. This doesn't affect:

Fixed a (never observed) race in the Async scheduler's closing of file
descriptors.

Previously, when the number of active system calls on an Fd.t
reached zero, the scheduler would call a closure that would
immediately schedule the close() system call in a thread. It was
possible (albeit very unlikely) that that close() would run before
the scheduler got a chance to update the epoll set, violating the
invariant that close is only ever called on fds not in the epoll
set.

Now, the scheduler enqueues an ordinary Async job to do the close,
and thus the close cannot happen until the next cycle, after the
scheduler has updated the epoll set.

Changed Reader to treat read returning EPIPE as end-of-file rather than
fail, to deal with OpenOnload.

This fixes an issue where reading from a TCP connection can return an
EPIPE if the tcp connection is immediately closed. This happens
when the application is running with onload and when the tcp
connection is closed immediately after creation.

Reduced allocation of the Async scheduler's File_descr_watcher, by using
callbacks to handle ready file descriptors.

Fixed In_thread.Helper_thread.create's error message if there are no
available threads in the thread pool.

The error message is now constructed eagerly. It had been constructed
lazily, so by the time it was rendered, the state might have changed,
possibly making threads available. This leads to a
nonsensical-looking error message that claims that there are no
available threads, immediately followed by a list of available
threads.

Moved Log from Async_extra to Async_unix, so that the scheduler can
refer to it.

When Writer.with_file_atomic is unable to clean up its temp file, raise
synchronously rather than asynchronously.

This eliminates complaints about an exception being thrown after a deferred
has been computed.

Added Log.rotate to force log rotation.

val rotate : t -> unit Deferred.t.

Fixed Log rotation to correctly reset the size and number of lines.

bignum

Fixed a bug in the =Zarith= library's to_float function.

These fixes first introduce tests from the base distribution, and then
backport a bugfix to the handling of to_float.

bin_prot

Switched build to use =config.h= rather than the command-line for
preprocessor variables.

Switched from ARCH_SIXTYFOUR to JSC_ARCH_SIXTYFOUR.

Fixed to support 32-bit integers, which are used in js_of_ocaml.

Do not make too many assumptions on integer size.
Integers are 32bit in Javascript.

Do not use the "get_float_offset" hack on 32bit
as it cannot be implemented in javascript.

core_extended

Added to Interval_map a more complete set of operations.

Removed Core_extended.Sexp.filter_record, which has been superseded by
Core.Std.Sexp.of_sexp_allow_extra_fields.

Added to Interval_map an Interval module, with the type of intervals used
in an interval map.

core_kernel

Added Float.int63_round_nearest_exn.

val int63_round_nearest_exn : t -> Core_int63.

Changed Hashtbl.sexp_of_t so that keys are sorted in increasing order.

This also applies to the sexp_of_t produced by Hashtbl.Make and
Make_binable. Sorting by key is nice when looking at output, as well as
in tests, so that the output is deterministic and so that diffs are
minimized when output changes.

Added to Info, Error, and Or_error a Stable.V2 module, whose bin_io
is the same as the unstable bin_io.

This makes it easy to force a use to be strict, which is sometimes
useful to accurately capture the state of a mutable data structure at
the time the error happens, lest it change by the time the error is
rendered.

Removed Interned_string module.

In Pooled_hashtbl, avoid trying to create arrays bigger than
Sys.max_array_length.

The problem affected 32-bit platforms.

Added Quickcheck module.

Supports automated testing with randomly-generated inputs in the style of
Haskell's Quickcheck library. Our adaptation supports flexible probability
distributions for values of a given type and uniqueness guarantees for
generated values.

Made Set.to_sequence and Set.split have the same interface as
Map.to_sequence and Map.split, respectively.

Fixed Float and Timing_wheel to compile on 32-bit platforms.

Added Lazy.Stable.V1.

Added List.reduce_balanced, which is like reduce, but relies on
associativity of f to make nesting of calls to f logarithmic rather than
linear in the input list length.

Added String_id.Make_without_pretty_printer.

Restricted Time_ns.Span values to be less than 135 years, which ensures the
corresponding floatTime.Span values have microsecond precision.

Fixed a Time_ns test that recently started failing due to crossing
the 135-year boundary.

Reducing the range of Time_ns.Span required adjusting the implementation
of Core.Time_ns.Option.Stable.V1, which (accidentally, incorrectly)
incorporated the (unstabilized) Core_kernel.Time_ns.Span.min_value as the
representation of bid_none and .max_value as ask_none. The prior
representation is preserved, but some previously allowed values are no
longer allowed and now raise exceptions!

Added Sequence.unfold_with_and_finish, a variant of unfold_with that can
continue the sequence after the inner sequence finishes.

Replaced Sequence.cycle with Sequence.cycle_list_exn, to work around a
bug in Sequence.cycle raising on the empty sequence.

Sequence.cycle can cause an infinite loop if its input is empty. It is
problematic to check whether the input sequence is empty.

* If we check it eagerly, we have to turn `cycle` into
`cycle_eagerly_exn`, and it will evaluate the first element twice.
* If we check it lazily, we might raise an exception in a seemingly
unrelated part of the code, and the usually-good habit of wrapping a
function like `cycle_exn` in `try .. with ..` would not catch it.

To get around these issues, [cycle] is changed to accept only lists as
inputs, not sequences. It is now called [cycle_list_exn].

Fixed assumptions about the size of integers, to support compiling to
Javascript, where integers are 32-bit.

Fixed build on Mac OSX.

Fix build when LINUX_EXT or TIMERFD are undefined.

Added Caml.Bytes.

Add an alias for Bytes in Caml. Fixes janestreet/core_kernel#46.

In Container, exposed polymorphic functions individually building container functions using fold or iter.

Exposed polymorphic functions in Core_kernel.Container for
individually building each of the Container functions using fold
or iter. E.g.:

async_extra

Added to Log a better mechanism for catching and handling background
errors, via set_on_error and an on_error argument to create.

Added Log.get_output : t -> Output.t list.

Changed Monitor.try_with so that errors after the initial return are
written to the global error log, rather than ignored.

Added Monitor.try_with_or_error and try_with_join_or_error.

try_with_or_error is intended to someday be renamed as try_with.
It also omits some of try_with's optional arguments: run and
rest. Different from try_with, try_with_or_error uses
~run:`Now, which we now believe is a more sensible behavior.

Fixed a bug in Versioned_typed_tcp that causes spurious and repeated
reconnects when user-level code disconnects.

Added Tcp.Server.create_sock, to create TCP servers that don't use
Reader and Writer.

Moved Async_extra.Rpc to its own library, Async_kernel_rpc, and
abstracted its transport layer.

Async_kernel_rpc depends only on Async_kernel. This allows
Async_rpc to be used in javascript or to try transports tuned for
different use cases. Versioned_rpc was moved to
Async_rpc_kernel as well.

Async_extra still provides an Rpc module with the Unix-dependent
part:

the Rpc.Transport module is augmented with
Async_unix.{Reader,Writer} based transports

the Rpc.Connection module is augmented with helpers for TCP
based connections

In sexp-formatted Log messages, output the sexp on a single line
rather than in multi-line "hum"an format.

This makes it possible to, among other things, easily grep
such logs.

Fixed a (very small) space leak in Persistent_rpc_client.

The fix was to use Deferred.choose and Deferred.choice instead
of Deferred.any and >>|. The old implementation added
a callback to the t.close_started ivar every time the connection
transitioned from connected to disconnected.

Added Persistent_rpc_client.create_generic, which is like create,
but generic in the function used to connect.

Fixed a race condition in the Versioned_typed_tcp interface that
caused a worker to miss a Connect message if the box is under high
load.

Query_client.create is called from Worker_impl.create in
a different async cycle than the following call to
Query_client.listen (really, Tail.collect under the hood) which
is made from Worker_impl.run.

When the load on the box is heavy (many workers starting and
connecting at the same time), the OS might take away the CPU from
the worker process between the two async cycles. The TCP socket
gets connected while the process is still waiting for its turn, and
eventually, when it's the worker's turn to grab the CPU, Async
scheduler might process the TCP event earlier than
Worker_impl.run.

Improved Udp.ready_iter to avoid intermediate exceptions by using
Syscall_result.

UDP loops use that, so will benefit.

Adjust the implementation slightly as well: made the inner loop
always exit on EAGAIN/EWOULDBLOCK to wait until ready, and give
other Async jobs a chance to run after EAGAIN/EWOULDBLOCK in the
outer loop.

Added Clock.Event.fired, and removed the fired value that was
returned by at and after.

val fired : t -> [ `Happened | `Aborted ] Deferred.t

Added Clock.Event.reschedule_{at,after}.

Fixed the space leak that caused nested Monitor.try_with to use
linear space rather than constant space.

Changed Monitor.try_with_ignored_exn_handling so that with
Eprintf` orRun f, the error processing runs in
Monitor.mainrather than in the monitor that called
Monitor.try_with`. This avoids space leaks due to chains of
monitors, e.g.:

There should be no change in behavior. This change was to make it
more clear that there is no intervening code that interferes with
the global backtrace state.

Made Deferred functions that take an argument
?how:[Parallel | Sequential ] accept
`Max_concurrent_jobs of int, which operates in a sequence in
parallel, limited via a throttle.

Made Deferred.Or_error match Applicative.S.

Fixed Scheduler.run_cycles_until_no_jobs_remain so that it continues
running if one has done Scheduler.yield.

Split the implementation of Deferred into a number of files, to
solve some problems with circularities.

Split into:

deferred.ml

deferred_sequence.ml

deferred_list.ml

deferred_array.ml

deferred_queue.ml

deferred_map.ml

deferred_result.ml

deferred_option.ml

For a sequence of multiple modules used to construct a module,
switched from the Raw_* prefix convention to the numeric suffix
convention. E.g. we now have Deferred0, Deferred1, Deferred.

async_parallel

Renamed Async_parallel as Async_parallel_deprecated; one should
use Rpc_parallel instead.

async_rpc_kernel

Moved Async_extra.Rpc to its own library, Async_kernel_rpc, and
abstracted its transport layer.

Async_kernel_rpc depends only on Async_kernel. This allows
Async_rpc to be used in javascript or to try transports tuned for
different use cases. Versioned_rpc was moved to
Async_rpc_kernel as well.

Added Rpc.One_way module, for RPCs that do not expect any response
or acknowledgment.

Sped up Rpc.Connection.

We have been able to send 6_000_000 (contentless) one-way messages
per second under some idealized circumstances.

In Rpc, added an optional heartbeat_config argument to configure
the heartbeat interval and timeout.

In Rpc, added some information to Connection_closed exceptions.

In Rpc.Implementations.create's on_unknown_rpc argument, added
a connection_state argument to the `Call variant.

When using Rpc.Connection.serve, one can put client addresses in
the connection_state, so this makes it possible to identify who
sent the unknown RPC instead of just saying what the RPC's name and
version are.

Added Rpc.One_way.Expert, which has an implement function that
gives direct access to the internal buffer instead of using a
bin-prot reader.

Made Rpc not raise an exception if a connection is closed due to
heartbeating.

Reworked Async_rpc_kernel's Transport implementation to not
require a transfer function similar to Async_unix.Writer.transfer.

Changed the RPC connection to flush the pipe when the writer is
closed, which was the only behavior of Async_unix.Writer.transfer
that had been relied on.

With this change, if someone closes the underlying writer by hand
the pipes won't be flushed, which should be expected anyway.

Fixed an issue where "normal" Pipe_rpc errors caused the connection
to shutdown.

Such errors include querying an unknown RPC or getting an exception
raised by the RPC implementation shutdown. Now such errors behave
like Rpc errors, i.e. they are completely ignored besides being
put in the return value of Pipe_rpc.dispatch.

Errors that occur later in Pipe_rpc still cause the connection to
close, since these should only occur if there is a bug somewhere.

One can now call close_reason : t -> Info.t Deferred.t to get the
reason the connection was closed.

Added One_way rpcs to Versioned_rpc.

Fixed Rpc to not write if the transport is closed.

The fix doesn't check that the writer is closed, but instead ensure
that we don't try to write when we shouldn't. This means that it
will still fail if the user close the transport by hand, which they
shouldn't do.

This change requires transport implementations to fail after they
have been closed. This is different from the semantics of
Async_unix.Writer, but the latter is non-intuitive and makes it
hard to check the correctness of the code. Moreover it is only
required for Async_unix.Writer.with_flushed_at_close which we
don't use.

async_ssl

Fix github issue #4 (some comments swapped).

async_unix

Made Unix.File_kind.t be Comparable, so it can be used in
<:test_result< >>.

Reduced allocation in Async's scheduler in the common path.

The allocation was in
Raw_scheduler.be_the_scheduler.compute_timeout, which was
(statistically, based on perf) the largest single allocator in one
of our applications. Now, it way down the list.

Note that the application is not a typical Async app in that it does
not sit in epoll very much, due to the way we do low-latency I/O.
This change will benefit everyone, but only a tiny bit.

Added Writer.write_bin_prot_no_size_header.

This is needed for Async RPC as it writes a different size on
its own.

Fixed a bug in Writer.transfer, which didn't close the pipe when the
consumer leaves.

Simplified the implementation of Writer.transfer:

replaced the big loop by a simple iteration function on the pipe
that just stop without filling its ivar when it sees the iteration
should stop for other reason that `Eof on the pipe: writer
closed, consumer left or stop requested by the user.

replaced the various choose by a single one and deal with the
closing reason only at this point.

Added Writer.close_started, symmetric to Writer.close_finished.

bignum

Upgraded from Zarith 1.2 to 1.3.

Removed dependence on Big_int.

bin_prot

Sped up bin_io of float array.

Bin_prot already had special fast handling for float array's but
with bin_io did not use it except for the special type
float_array. Now, there is fast handling for float array and
its aliases, for example price array when type price = float.

Changed Size.bin_size_array, Write.bin_write_array and
Read.bin_read_array short circuit to the fast path when it detects
that float array is handled. Each of these functions receives
a function for handling array elements and short circuits when the
function for handling elements is equal to the function for handling
floats, using physical equality of closures.

To cause short circuiting for aliases of float, changed bin_io
so that aliased bin_io functions are equal the the bin_io
functions of the original type. That is an optimization for itself
regardless whether it's used for float. Before this change, every
function generated for aliases were eta-expanded leading to
different closures at runtime for each type.

Short circuiting needs to apply to the handling function rather than
to the value at hand because:

the value is available only in size and write, and we need
a way to make read work as well.

even when the value is a float at runtime, the handling of
a specific float alias may have been overridden by a custom one.

Made a slight improvement to bin_read_float_array: since the array
is going to be filled with read values, there is no need to fill it
with 0. after allocation:

Made Unix.Cidr match Comparable.S_binable, and added Cidr.create and
Cidr.netmask_of_bits.

Moved Unix.tm and Unix.strftime from Core_kernel to Core.

Made Crc.crc32 return Int63.t rather than int64, and added
Crc.bigstring_crc32 and Iobuf.crc32.

Cleaned up old cruft in the C stubs for CRC checking.

Added Iobuf.narrow_lo and narrow_hi, which comprise
Iobuf.narrow.

Changed Linux_ext.Timerfd, Epoll.wait, and Unix.select to use
(int) Time_ns rather than (float) Time.

This avoids spurious float conversions and rounding problems.

Made all timeouts consistently treat negative timeouts as "timeout
immediately".

This fixes an incorrect behavior of Linux_ext.Timerfd.set_after
and set, which had been rounding to the nearest microsecond, which
was particularly bad for time spans smaller than 500ns, which would
be rounded to zero, and then would cause the timerfd to never fire.
Now, the small span is directly fed to timerfd_settime. We also
changed a span of zero to be treated as 1ns, to avoid the behavior
of timerfd_settime 0, which causes the timerfd to be cleared and
never fire.

Made Or_error match Applicative.S.

Added Command.Param, with the intention of one day replacing
Command.Spec and providing an applicative interface for command-line
parsing.

This change required lots of rearrangement of command.mli so that
Command.Param and Command.Spec could share large portions of
their interface. As a side effect, the interface is more sweeksy
than before.

Added Command.shape, for exposing the shape of a command, including
what subcommands its has.

Changed Syscall_result.to_result to return a preallocated object,
and changed many uses to take advantage of this property.

Pattern matching on results is much clearer than if-analysis and
avoids double-checking errors in many cases.

Syscall_result.to_result can only return preallocated results for
a few Ok values from Syscall_result.Int, of course, and likewise
for other large ok_value types. We have initially, and
arbitrarily, limited preallocation to 64 errno's and 2048
ok_value's.

Added Sys.big_endian : bool, from Caml.Sys.

Disabled unit tests in Time_ns that started failing around 10:40pm
NYC 2015-05-15.

The tests indicate an off-by-one-microsecond error in round tripping
between Time.Span.t and Time_ns.Span.t.

core_bench

Exposed the equality of Core_bench.Std.Bench.Test.t with
Core_bench.Test.t, so that one can get the name of a test.

This is useful for filtering based on test name.

core_extended

Removed the Stats_reporting module.

Renamed Quickcheck module to Quickcheck_deprecated. It's replaced
by Janecheck, which for now is a separate library in the core_extended
package, but will soon be merged into core.

Moved the Selector module to its own library. This is for internal
reasons related for code review; it is included as a library within the
core_extended package for now, but may move to another home in the
future.

Made Timing_wheel_ns keys be Int63.t rather than int, so that
behavior is consistent on 32-bit and 64-bit machines.

Also, made Timing_wheel.Interval_num an abstract type.

Hid the bytes type in Core.Std, so that type errors refer to
string rather than bytes.

Added Bytes module so that people can say Bytes.t if they
need to.

Now we get reasonable error messages:

String.length 13
-->
Error: This expression has type int but an expression was expected of type
string
"" + 13
-->
Error: This expression has type string but an expression was expected of type
int

Modernized the coding style in Timing_wheel.

Replaced Unpack_buffer.unpack with unpack_into and unpack_iter,
to avoid allocation.

Unpack_buffer.unpack created a (vector-backed) Core.Std.Queue
for each call. When unpacking a buffer containing many values,
resizing of the buffer can be costly and in some cases leads to
promotions of short-lived data to the major heap.

Add a flag -sandbox-action to jenga build to run the action in an
environment that (attempts to) detect missing dependencies or overly
large sets of targets.

Don't interpret paths of shape ./foo.ext as aliases.

Always interpret Dep.path path as the file at path rather than the
default alias for the directory at path

Make jenga look for the variable JENGA_OPTIONS, and use it to add
a debug setting.

Fix handling of directories created during the build.

Basic support for symlink resolution.

Add an option to the monitor command to only display a single snapshot
of the progress, rather than a continually updated pipe.

Fix a deadlock where File_access throttle and directory lock are
obtained in the wrong order.

ocaml_plugin

In copy_source_files_to_working_dir, exclude files that start with
a dot.

emacs creates temporary files that cannot be read with names like
.#foo.ml, and attempting to copy those causes this function to
fail.

pa_ounit

Made Pa_ounit_runtime not depend on OUnit anymore.

The dependency hasn't been needed in more than 2 years.

re2

Fixed a bug in Re2.find_all_exn, extant since 2014-01-23, in which
it returns spurious extra matches.

Using pattern b and input aaaaaaaaaaaab is expected to return
a single match at the end of the input but instead returned the
match multiple times, approximately as many times as
input length / min(match length, 1).

Added tests for this function and also get_matches which uses the
same code.

Updated to new version of upstream library.

sexplib

Inline some calls that js_of_ocaml was unable to recognise as
tail-recursive (cf. issue #14)

112.24.00

async

Keep up to date with interface changes in Async_kernel, Async_extra and
Async_unix.

async_extended

async_extra

Changed Log to not eagerly run the rotation loop when an
Output.Rotating_file is created.

Changed Log.Output.combine to write log outputs in sequence rather than
parallel, to give the correct semantics when the same output is included
multiple times in Log.create.

This fixed a test that was failing in lib_test/log_test.ml.

Remove Log.Rotation.t_of_sexp.

Made Command.async* functions flush stdout and stderr before calling
shutdown, to avoid timeouts causing data to be dropped.

For now, we're making this change in Command rather than Writer.
Writer already has at_shutdown handlers. We've observed that they
don't behave well for command-line programs w.r.t. stderr. So, the
thinking of this feature is to try out a different at_shutdown
behavior, just for Command executables and just for stdout and
stderr. If it works out, maybe we move it into Writer proper.
Putting the change in Command for now reduces the scope of what is
affected by the experiment, and hopefully correlates well with where
the change is likely to help.

Fixed a race in Versioned_typed_tcp in which a message can be dropped
between Server.create and Server.listen

Simplified the implementation of Rpc.

Rpc has an internal Response_handler module, which is just a record containing
a response-handling function and an already_removed boolean field. It turns
out that this is unnecessary: already_removed is set to true when the
function returns remove`, but if it returnsremove` then it will also be
removed from a hash table, and we only call the function immediately after
looking it up in that hash table.

This wasn't always pointless: this function used to return deferred values and
run inside a throttle. The simplification is only possible because we made it
synchronous a while ago.

Added Tcp.Server.num_connections function.

Added creation functions for Versioned_rpc menus, for use in clients of an
RPC proxy.

These are for use in clients of an RPC proxy, which can't use the
regular menu mechanism since they each need to have many menus (one
for each potential target) but should only need to have one connection
(to the proxy).

Added to Rpc expert submodules, Implementations.Expert and Rpc.Expert,
with low-level access for implementing a proxy that can handle queries without
knowing their names, types, etc. in advance.

Renamed Rpc.Implementation.Description as Rpc.Description.

Added Rpc.{Rpc,Pipe_rpc,State_rpc}.description accessor functions.

Added Rpc.Implementation.descriptions, which returns all RPCs in an
Rpc.Implementations.t.
This was needed for the rpc_discovery library: given an Implementations.t we want to advertise at

prefix/<rpc_name>/<rpc_version>/host_and_port = <host_and_port>

Added combinators to Rpc.Implementations: lift, add, add_exn.

async_kernel

Now depends on Core_kernel instead of Core.

Async_kernel.Clock uses Core_kernel.Time_ns and
Core_kernel.Timing_wheel_ns rather than Core.Time and
Core.Timing_wheel_float.

Added Async_kernel.Types module to deal with the mutual recrsion of
Async_kernel's types.

This should help eliminate the complexity and make it easier to make changes
without running into as many constraints due to module/type ordering.

Merged Types.Jobs into Types.Scheduler.

Improved the performance of Deferred.bind, eliminating an allocation in
Ivar.connect.

Time_stamp_counter unit test has an 18s unit test, which seems
excessive. Take a couple of orders of magnitude off the number of
iterations.

Added Time_ns.pause, whose implementation is the same as Time.pause.

This involved moving the nanosleep C code from Core to
Core_kernel.

This was necessary so that Async_kernel can pause without introducing
a dependence of Async on Core.

Made Core_kernel.Time_ns.Alternate_sexp use a similar format to Core.Time_ns.

This was needed so that Async_kernel can use a nice sexp format for
time spans.

Changed Timing_wheel implementation to use Time_ns, and moved to
Core_kernel.Timing_wheel_ns; made Core.Timing_wheel a wrapper around
Timing_wheel_ns.

Generalized the timing-wheel interface to be parametric in Time, so
that one interface applies to both Timing_wheel and
Timing_wheel_ns.

Generalized the timing-wheel unit tests to a functor,
Timing_wheel_unit_tests.Make, that is used to test both
Timing_wheel_ns and Timing_wheel_float. Moved a few tests that
depend on Time and Date from the functor into
timing_wheel_float_unit_tests.ml.

Split out Timing_wheel.Debug into a separate functor,
Timing_wheel_debug.Make.

This was done in so that Async_kernel can depend only on
Core_kernel and not Core.

custom_printf

Added syntax to use Sexp.to_string_mach rather than Sexp.to_string_hum.

Added syntax %{sexp#mach:<type>}, which is like %{sexp:<type>},
except it calls Sexplib.Sexp.to_string_mach instead of
Sexplib.Sexp.to_string_hum.

In fact, you can put any identifier after the sexp# and it will call
Sexplib.Sexp.to_string_<that identifier>; however, there are no other
such functions right now.

jenga

Interns strings, significantly reducing memory use.

ocaml_plugin

Minor update: follow Async evolution.

pa_ounit

Added -verbose switch to time unit tests, so we can easily see which ones
are taking a long time.

pa_test

Reduce code generated by pa_test in favor of more code in pa_test_lib.

Less generated code means less time spent compiling it.

Reducing code in pa_test.ml is good, because:
a) Staging semantics make this code hard to understand.
b) This code is written using the less familiar revised OCaml syntax.
b) We get less code to migrate to syntax extensions.

Re-used location code in pa_here; side benefit that we get full path names
in (Loc...) instead of just the file's basename.

async_extra

In Versioned_typed_tcp, check that the writer on the other side is
not closed in the Pass_on case.

Added a new way to implement an RPC, where the implementation
doesn't return a deferred.

This "blocking" rpc implementation guarantees that the rpc will in
fact be fully dispatched by the time the implementation returns.

This can be used to skip the deserialization of the query, and
instead operate directly in the message contents as received.

Also, fixed a bug in which the query handler (and therefore the
connection state) was being called before the internal async rpc
handshake was finished.

Added an optional job_tag argument to Sequencer_table.enqueue,
to display for debugging.

Added an optional argument to TCP-connection functions to control
the local interface used to connect.

To implement this this, extended Tcp.connect to work on a bound socket.

Added with compare to Process.Output.t.

Added Process.Output.Stable module.

Exposed concrete rpc in Versioned_rpc.Both_convert.

Changed Cpu_usage to take its first sample after waiting, rather
than immediately.

This fixes a problem where the first sample could be NAN or Inf.

Made Log buffer-age be unlimited, to avoid exceptions when log
writes are blocked for long periods.

Improved Log.t_of_sexp's error message.

Changed Rpc.Connection.client and with_client to raise some
errors which had been dropped during RPC dispatch.

Previously, errors dispatching Rpc.Rpc.t's were handled correctly
and returned or raised by the relevant dispatch functions. However,
errors that occurred in the middle of handling a Rpc.Pipe_rpc.t or
Rpc.State_rpc.t were swallowed. This is because they happen after
the dispatch functions have returned, and the dispatch interface
doesn't allow for errors to occur in the middle of the pipe -- they
must be raised to the monitor in effect when the Rpc.Connection.t
is created. Errors could be raised to the effective monitor at the
dispatch call, but the failure causes the entire connection to go
into error, so the connection's monitor seems more appropriate.
These errors weren't propagated to the caller because client and
with_client both used Monitor.try_with without rest handling,
causing /any/ errors caused while handling the connection (after the
Connection.t has been returned to the user) to be dropped.

In Rpc, exposed some optional parameters from the Tcp module:
?max_pending_connections and ?buffer_age_limit.

async_kernel

Fixed a space leak in Clock.Event.abort, making it free the job
that was created and stored in the timing wheel.

Moved Scheduler.yield from Async_unix.

Fixed a bug in Scheduler.yield_every, so that it doesn't
initialize the scheduler until the staged function is called.

Changed Pipe.merge to check whether its output is closed, and if
so, stop rather than write to it (which raised).

Also, made Pipe.merge close its inputs whenever its output is
closed.

Changed Clock.at to return Deferred.unit if it is supplied a
time in the past.

Previously, it would create an empty ivar and a job to fill it that
would run in the next cycle.

Changed Clock.Event.status to return Will_happen_at of
Time.t``` rather thanWaiting```, if applicable.

Added Ivar.create_full.

Moved the use of Linux_ext to Async_unix.

This is one of the necessary steps in making Async_kernel depend
on Core_kernel rather than Core.

async_parallel

Modernize the code

async_smtp

Moved from janestreet-alpha

async_ssl

moved ffi_bindings and ffi_stubgen in separate libraries

async_unix

Moved Scheduler.yield to Async_kernel.

Added Reader.load_annotated_sexp* functions.

These are like the existing Reader.load_sexp* functions, except
they return annotated sexps rather than sexps. Having annotated
sexps is useful so that one can report error positions to the user
when processing values built by the t_of_sexp_ functions /after/
they return. I.e. when there aren't syntax errors in the sexps, but
rather semantic errors detected later.

Removed noise and redundancy from Reader.load_sexp error messages.

Added Writer.save_sexps, analogous to Reader.load_sexps.

Made Writer errors raised by the background flush job include the
entire Writer.t, rather than just the Fd.t.

Added to Writer.transfer an optional argument to limit the number
of values read at once from the pipe.

The old behavior is to have no limit and remains the default.

Added to Writer some missing checks for functions that should
ensure the input writer isn't closed.

Changed Scheduler.run_cycles_until_no_jobs_remain to pause so that
alarms scheduled to fire in the past actually fire.

This is necessary because of the timing-wheel, which doesn't
guarantee to fire an event until alarm-precision after it was
scheduled.

Without this change, some tests unexpectedly fail, due to jobs not
running that should have.

Then we have that A.t <> B.t but C.t = D.t. This can lead to
subtle bugs, e.g. Unique_id.Int (Unit). Note that it is perfectly
valid to apply any module to F, even though that is certainly not
what we want.

In 4.02, we can explicitly say that functor generates new types,
i.e. it is generative. For this we use argument (). So F
becomes

moduleF() : sigtype t end

You can only apply F to () or (struct end) but each
application yields a new type t.

(** [prepend a ~f] returns a value [x] such that each call to [x.return] first applies [f] before applying [a.return]. The call to [f] is "prepended" to the call to the original [a.return]. A possible use case is to hand [x] over to an other function which returns ['b] a subtype of ['a], or to capture a common transformation [f] applied to returned values at several call sites. *)valprepend : 'areturn -> f:('b-> 'a) -> 'breturn

Moved the Gc module's alarm functionality into a new
Gc.Expert.Alarm module.

ocaml_plugin

When a plugin cache directory is used by several executables with
incompatible cmis/compilers, and the cache config option
try_old_cache_with_new_exec is set to true, this could lead to the
following error:

Plugin failed: (ocaml_dynloader.ml.Dynlink_error"interface mismatch")

This feature fixes this.

Since it modifies some record, for later changes it seems easier and
more conservative to allow field additions without breaking older
version. Thus we allow extra fields in persisted records.

let t_of_sexp =Sexp.of_sexp_allow_extra_fields t_of_sexp

New executables can read both old and new caches, but old
executables will either blow away new caches, or if the config says
the cache is read-only, fail.

Take the chance to modernize part of the code.

Switched tests to unified tests.

Fixed bugs dealing with paths with spaces in them.

Check that plugins have the expected type before running them rather
than after, which is what one would expect.

Also check that runtime and compile types match in
check_ocaml_src_files and
compile_ocaml_src_files_into_cmxs_file.

pa_ounit

only spend time to format test description when tests are run

patdiff

The call to Pcre.full_split in patdiff_core.ml rely on a bug of
pcre-ocaml <= 7.1.2.

To get the same behavior with pcre-ocaml >= 7.1.3 we need to pass
~max:(-1).

rpc_parallel

sexplib

Added sexp_of_ support for GADTs, and remove the not-quite-working
support for of_sexp.

textutils

Added a ~narrow argument to Text_graph.render

typerep

Split out typerep_extended which is now using core_kernel

typerep_extended

typerep_extended now use core_kernel

112.06.00

async_extended

Unwound a recent change to Mailbox where one invocation of receive
would put aside items, preventing other invocations from noticing
them.

Added Delimited.Row.nth_conv_exn, as a counterpart to
get_conv_exn.

Fixed File_updates handling of identical mtimes.

async_extra

In Log, exposed the raw message.

Changed Rpc creators' connection_state to be a function that takes
the connection and returns the state.

This makes it possible for the connection state to actually get
a handle on the connection itself, which simplifies a number of
idioms for using RPC. In particular, it makes it easier to respond
with an RPC back to a client over client's own connection.

Fixed some nondeterministically failing tests.

In Log, made logs discard messages when their output list is empty.

Also, removed redundant tracking of current level.

Moved Udp.bind_to_interface_exn to Unix module in async_unix.

Added Versioned_typed_tcp.Repeater.

Repeater is used in the cases where we want to inspect and possibly
alter the flow between a client and a server without having to
change either the client or the server or the protocol between them.

async_kernel

Added Deferred.Sequence module, analogous to Deferred.List but for
Core_kernel.Std.Sequence.

Modernized code style.

async_unix

Added Writer.behave_nicely_in_pipeline, which makes a program behave
nicely when used in a shell pipeline where the consumer goes away.

Modernized code style.

Removed spurious overrides in Signal: set and signal.

These overrides are no longer necessary because the functions moved
from Core.Signal to Core.Signal.Expert.

Moved async_extra's Udp.bind_to_interface_exn to Unix.

bignum

Added functions to round from Bignum.t to Bigint.t, and to convert
Bigint.t into Bignum.t.

bin_prot

Sped up float and float array operations.

Removed a use of Obj.magic in code generated by pa_bin_prot for
polymorphic variants that led to memory unsafety.

Previously, pa_bin_prot generated this kind of code for
polymorphic variants:

Made Ocaml_dynloader.find_dependencies also support files with
shebang lines.

pa_bench

Made the code generated by pa_bench for BENCH not use ignore,
because OCaml 4.02 will remove dead code in some cases, meaning the
benchmarks are no longer measuring what they should. Instead the ignore
is deep inside Core_bench, which is likely out of reach of the
compiler.

The result of the user functions given to BENCH_FUN and
BENCH_INDEXED are changed so they don't have to return unit and
people are encouraged not to use ignore when these functions don't
return unit (you will get the same warning though, i.e. a warning
if the result of your function is a function too, thus preventing
unintended partial applications).

For example, here are a few benchmarks and their output before the
fix:

Many of the the numbers above are much lower than they should be
because of the implicit ignores inserted by the benchmark caused the
division to to eliminated by the compiler. After the fix, the same
benchmarks produced more meaningful numbers:

sexplib

Improved the implementation of Exn.sexp_of_t, using the unique id in
exceptions in OCaml 4.02.

We use the identifier to map exception constructors to converters.

112.01.00

async

update tests

async_extended

Clarified an error in Rpc_proxy.

async_extra

Changed Persistent_rpc_client.connected to avoid returning a
connection that is closed at the time it was called.

Optimized Rpc.implement so that if a server's implementation
returns a determined deferred, then the output is immediately
serialized and written out for the client.

This reduces memory consumption, improves throughput and latency.
Measurements with the pipe_rpc_test program showed that a server
went from processing 600_000 msg/sec, to 2_200_000 msg/sec before
pegging the CPU.

Changed Log's output processor's batch size from 1_000 to 100.

Added Persistent_rpc_client.close and close_finished.

In Rpc.Connection.client and with_client, used the
handshake_timeout as the timeout passed to Tcp.connect.

handshake_timeout was previously used only for the Rpc module's
handshake timeout.

Changed Writer.write_bin_prot to use the new
Bigstring.write_bin_prot.

bignum

Added Bignum.Bigint module, with arbitrary-precision integers
based on Zarith, which is significantly faster than the
Num.Big_int library.

bin_prot

In Write, improved some OCaml macros to name values and avoid
calling C functions multiple times.

core

Removed vestigial code supporting OCaml 4.00.

Added Command support for flags that are passed one or more times.

Added Command.Spec.one_or_more and
Command.Spec.non_empty_sequence to deal with the cases where you
expect a flag or anonymous argument (respectively) to be passed one
or (optionally) more times. This is common enough and distinct from
the case where you want the argument passed zero or more times that
it seems like we should canonize it in the library.

In Lock_file, made stale lock detection more robust.

Made Lock_file.create foo succeed if foo is absent and
foo.nfs_lock file is present and stale. Previously, it would
fail.

Added Filename operators /^ and /@, and of_parts, like the
same functions for Catalog paths.

Changed Iobuf functions that advance the iobuf to not also return
a redundant number of bytes processed.

This avoids a small allocation (in the case of the int option
functions) and normalizes the result (so the same information isn't
returned two ways). Actually, it doesn't yet avoid the allocation in
the implementation, as the corresponding Bigstring functions must
still return the number of bytes processed, and currently do so as an
option. We hope to eventually change that.

In the future I expect we will change unit to some error variant
to also avoid the exception construction for EWOULDBLOCK/EAGAIN. We
can even make Unix syscalls noalloc if we're careful.

In Unix module, added unit tests for Cidr.does_match.

core_bench

fixed legacy format string

core_extended

Added Float_ref module, which is like float ref but faster for
sets due to bypassing the write barrier.

Added Bin_io_utils.Wrapped.t, which defines an 'a t with bin_io
that supports size-prefixed serialization and deserialization.

Wrapped has two useful submodules, Opaque and Ignored, for
efficient handling of size-prefixed bin-io values in cases where
serialization can be bypassed. See the comments in the module for
more details.

core_kernel

Removed vestigial code supporting OCaml 4.00.

Used {Hashable,Comparable}.S_binable in Day_of_week and Month.

Improved the performance of Set_once.set.

Added Type_equal.Lift3 functor.

Replaced occurrences of Obj.magic 0 with Obj.magic None.

With the former the compiler might think the destination type is
always an integer and instruct the GC to ignore references to such
values. The latter doesn't have this problem as options are not
always integers.

This module supersedes the read_only, read_write, and
immutable phantom types, which are now deprecated, and will be
removed in the future. This module uses a different approach using
sets of polymorphic variants as capabilities, and contravariant
subtyping to express dropping capabilities.

This approach fixes a bug with the current phantom types used for
Ref.Permissioned in which immutable types aren't guaranteed to
be immutable:

The bug stems from the fact that the phantom-type parameter is
covariant, which allows OCaml's relaxed value restriction to kick
in, which allows one to create a polymorphic value, which can then
be viewed as both immutable and read write. Here's a small
standalone example to demonstrate:

The new approach fixes the problem by making the phantom-type
parameter contravariant, and using polymorphic variants as
capabilities to represent what operations are allowed.
Contravariance allows one to drop capabilities, but not add them.

Added Int.Hex module, which has hexadecimal sexp/string
conversions.

Added Gc.major_plus_minor_words, for performance reasons.

core_profiler

fixed legacy format string

custom_printf

Fixed uses of printf=-styleformat strings that have unspecified
behavior in OCaml 4.02 and will become errors.

jenga

Made standalone actions atomic, just like actions associated with
target files.

Running actions and recording the result in the persistent
.jenga.db should be performed atomically for standalone actions,
as it is for actions which are associated with target files

ocaml_plugin

Changed to not use rm -r when it is expected to remove one file.

rpc_parallel

Initial import.

sexplib

Replaced occurrences of Obj.magic 0 with Obj.magic None.

With the former the compiler might think the destination type is
always an integer and instruct the GC to ignore references to such
values. The latter doesn't have this problem as options are not
always integers.

textutils

Added Ascii_table.Table_char, to expose the special table border
characters.

type_conv

Updated ast matching for 4.02

111.31.00

jenga

Switched API to composable generator schemes.

Support -api flag to show the embedded API.

New examples.

111.28.00

async_extra

Added to Versioned_rpc a non-functor interface.

Added Log.level, which returns the last level passed to set_level.

Enabled Async-RPC pushback in the Tcp_file protocol.

async_unix

Added Shutdown.set_default_force, which allows one to change the
default force value used by shutdown.

Added Command.exec, which allows one to include the Command
hierarchy from one executable in another.

Command.exec takes the file path to an executable that uses the
Command module and returns a Command.t that integrates the
executable (by exec'ing it), including providing recursive help and
autocompletion as if it were a standard Command.t.

Replaced most uses of Hashtbl.replace with Hashtbl.set.

Renamed Float.epsilon to robust_comparison_tolerance, to avoid
confusion with epsilon_float.

core_extended

Implemented Int.gcd using binary GCD in C, for improved performance.

Added Bin_io_utils.Serialized, which stores a value in memory as its
bin-io representation.

Writing such a value just blits the value.

Moved Text_block from Core_extended into Textutils.

Added modules Hashtbl2 and Hashtbl2_pair.

core_kernel

Added Pooled_hashtbl.resize function, to allow preallocating a table
of the desired size, to avoid growth at an undesirable time.

Added Pooled_hashtbl.on_grow callback, to get information about
hashtbl growth.

Changed Hashable.Make to not export a Hashable module.

The Hashable module previously exported was useless, and shadowed
Core.Std.Hashable.

Moved Common.does_raise to Exn.does_raise, to make it easier to
find.

Added Float.one, minus_one, and ~-. (fixes #12).

Removed Core.Std.unimplemented and renamed it as
Or_error.unimplemented.

It is not used enough to live in the global namespace.

jenga

Fixed problem that caused rule failed to generate targets.

ocaml_plugin

Fixed a bug in tests that could leave the repository in a state where
running the tests would fail.

The bug happened if the tests were interrupted after creating
read-only directories but before cleaning then up.

Made Core_hashtbl_intf more flexible. For instance supports
modules that require typereps to be passed when creating a table.

Address the following issues:

The type ('a, 'b, 'z) create_options needs to be consistently used
so that b corresponds with the type of data values in the returned
hash table. The type argument was wrong in several cases.

Added the type ('a, 'z) map_options to Accessors so that
map-like functions -- those that output hash tables of a different
type than they input -- can allow additional arguments.

Fixed a bug in Dequeue's bin_prot implementation that caused it
to raise when deserializing an empty dequeue.

Made Container.Make's interface match Monad.Make.

Deprecated infix or in favor of ||.

Simplified the interface of Arg (which was already deprecated in
favor of Command).

Replaced Bag.fold_elt with Bag.filter.

Memo.general now raises on non-positive cache_size_bound.

Removed Option.apply.

Removed Result.call, Result.apply.

Moved Quichcheck to core_extended.

It should not be used in new code.

custom_printf

Fix 4.02 compatibility.

jenga

Switched to un-version-numbered API.

Renamed Tenacious_sample_lib.Tenacious to
Tenacious_sample_lib.Tenacious_sample to avoid conflicts in the
public release.

Write buildable_targets.list (on alias .info).

ocaml_plugin

ignore more warnings by default

patdiff

add a ?file_names argument to Compare_core.diff_strings

patience_diff

refactoring and more unit tests

sexplib

Fix compatibility with OCaml 4.02

111.21.00

async_extra

Added Sexp_humLog.Output.format, which is useful for making logs
more human readable.

Added with compare to Rpc.Implementation.Description.

async_ssl

Upgraded to use new ctypes and its new stub generation methods.

async_unix

Added Process.wait_stdout and wait_stdout_lines, which are like
run and run_lines, but take a Process.t.

core

Fixed an issue where Time.Zone.init would not properly traverse the
directory containing timezone information.

Added Time.With_utc_sexp, which has stable serialization of Time.t
that is byte-for-byte equal across timezones.

Made Uuid stable.

Made Piecewise_linear stable.

core_kernel

Removed our custom C stub for closing channels, reverting to the one
in the OCaml runtime.

A long time ago we found that the OCaml runtime did not release the
lock before calling close on the fd underlying a channel. On some
filesystems (e.g. smb, nfs) this could cause a runtime hang. We
filed a bug with INRIA and wrote our own close function which
In_channel calls to this day. The bug has long been fixed, and
our function is probably buggy, so this reverts us to the runtime's
close.

Added Float.{of,to}_int64_preserve_order, which implement the
order-preserving zero-preserving bijection between non-NaN floats and
99.95% of Int64's.

patdiff

(** Iter along the lines of the diff and the breaks between hunks. Offers more flexibility
regarding what the caller wants to do with the lines *)
val iter_ansi
: f_hunk_break:((int*int) -> (int*int) -> unit)
-> f_line:(string -> unit)
-> string Patience_diff.Hunk.t list
-> unit

patience_diff

Added plain differ Plain_diff and use it in some cases for
improved results.

Move modules under Patience_diff_lib.Std.

111.17.00

async_extra

Added module Persistent_rpc_client, an RPC client that attempts to
reconnect when the connection is lost, until a new connection is
established.

Significantly sped up the Rpc module by removing Bigstring
serialization.

Performance of the two implementations was tested by building a
simple client/server executable that would count major cycles.
Sending 100 byte messages at a rate of 50k/second shows (on both
sides of the RPC):

original:

~160 major cycles in 30s

CPU usage around 60%

new:

~10 major cycles in 30s

CPU usage <= 2%

Enabled a version of Pipe_rpc and State_rpc where the consumer
can pushback on the producer if it can't consume the contents of the
pipe fast enough.

Added Command.async_or_error and renamed Command.async_basic to
Command.async, leaving async_basic a deprecated alias for the
new name.

Command.async_or_error is similar to Command.basic and
Command.async, but accepts a unit Or_error.t Deferred.t type.

Added Persistent_rpc_connection.current_connection, so that one
can detect whether one is currently connected.

valcurrent_connection : t -> Rpc.Connection.toption

async_inotify

Upgraded library to use inotify 2.0

async_kernel

Renamed Monitor.errors to Monitor.detach_and_get_error_stream
and Monitor.error as Monitor.get_next_error.

The use of detach in the name is intended to make clear that
errors do not propagate to the parent.

Added several other non-stream =Monitor= functions to capture common
use cases of Monitor.detach_and_get_error_stream:

detach_and_get_next_error
detach_and_iter_errors
detach

bignum

Improved the performance of binprot deserialization by removing the
allocation of an intermediate type.

core

Fixed a bug in Bigstring.really_recv if recv doesn't receive all
the data it wants.

This bug has been around forever; it may not have caused trouble
because Bigstring.really_recv (1) is barely used (the only use is
in Bigstring.unmarshal_from_sock) and (2) passes recv the
MSG_WAITALL flag, so it will read the full amount unless it gets
interrupted by a signal.

Fixed Bigstring.read's handling of EINTR so that it retries
rather than returning zero.

This fixes a bug introduced in 111.09 in the interaction between
Bigstring.read and Async.Reader. Prior to 111.09,
Bigstring.read would raise on EINTR, and Async.Reader would
propagate the exception. From 111.09 to 111.16, Bigstring.read
would return zero, which would confuse Async.Reader into thinking
it reached EOF when it hadn't. From 111.17, Bigstring.read will
retry and not return zero when not at EOF.

We believe the bug was rare, because otherwise we would have
frequently seen EINTR exceptions prior to 111.09.

Added Command.Spec.apply and pair, which allow one to program
more with Spec.param rather than Spec.t.

Added Command.Spec.file, which builds an Arg_type value with the
same autocompletion as Spec.file.

(** [file] defines an [Arg_type.t] that completes in the same way as [Command.Spec.file], but perhaps with a different type than [string] or with an autocompletion key. *)
val file
: ?key:'a Univ_map.Multi.Key.t
-> (string -> 'a)
-> 'a t

core_extended

Added some functions to Splay_tree:

length

keys

data

to_alist

delete_{after,before}

map

map_range

split.

core_kernel

In Bigstring, made many operations use compiler primitives new in
OCaml 4.01.

Exposed Bigstring.get and set as compiler primitives in the
interface.

Previously, many Dequeue.dequeue-type functions worked by raising
and then catching an exception when the dequeue is empty. This is
much slower than just testing for emptiness, which is what the code
now does.

This improves the performance of Async.Writer, which uses
Dequeue.dequeue_front.

patdiff

Removed latex output.

patience_diff

Exposed Patience_diff.matches.

sexplib

Make the camlp4 dependency optional

111.13.00

async_extra

For Typed_tcp.create, added a Client_id.t argument to the auth
callback.

async_unix

Added Scheduler.yield_every, which returns a function that calls
Scheduler.yield every n-th call.

This is useful in circumstances where you don't have strict control
over where a deferred is examined, as in a Deferred.List.iter.

bignum

Eliminated the dependence of Bignum on Re2, and reduced dependence
from Core to Core_kernel.

Extended the rounding interface to bring it in line with int and float
rounding.

Improved the performance of Bignum's binprot.

Bignum's binprot had been to just binprot the decimal string
representation. This is both slow to do and unnecessarily big in
the majority of cases. Did something better in the majority of
cases and fell back to this representation in the exceptional case.

Added a new function, Binary_searchable.binary_search_segmented,
that can search an array consisting of two segments, rather than ordered
by compare.

(** [binary_search_segmented ?pos ?len t ~segment_of which] takes an [segment_of]
function that divides [t] into two (possibly empty) segments:
{v
| segment_of elt = `Left | segment_of elt = `Right |
v}
[binary_search_segmented] returns the index of the element on the boundary of the
segments as specified by [which]: [`Last_on_left] yields the index of the last
element of the left segment, while [`First_on_right] yields the index of the first
element of the right segment. It returns [None] if the segment is empty.
By default, [binary_search] searches the entire [t]. One can supply [?pos] or
[?len] to search a slice of [t].
[binary_search_segmented] does not check that [segment_of] segments [t] as in the
diagram, and behavior is unspecified if [segment_of] doesn't segment [t]. Behavior
is also unspecified if [segment_of] mutates [t]. *)
val binary_search_segmented
: ?pos:int
-> ?len:int
-> t
-> segment_of:(elt -> [ `Left | `Right ])
-> [ `Last_on_left | `First_on_right ]
-> int option

Made Queue match Binary_searchable.S1.

Made Gc.Stat and Gc.Control match Comparable.

Fixed some unit tests in Type_immediacy that were fragile due to GC.

patience_diff

Moved Patience_diff out of Core_extended into its own library
depending only on Core_kernel.

sexplib

In Sexplib.Std, renamed Macro as Sexp_macro.

type_conv

Removed some unused-value warnings when with is used in signatures.

Removed warnings in cases like:

include (module_expr : sig type t with bin_io end)

111.11.00

async

Updated the sound.ml example

async_extra

Made Log more fair with respect to other Async jobs, by working on
fixed-length groups of incoming log messages.

Previously, Log had processed everything available. The change
gives other Async jobs more of a chance to run.

async_kernel

Added Clock.run_at_intervals, which runs a job at regular
intervals.

async_unix

Improved the behavior of the scheduler's thread pool when
Thread.create raises.

With this improvement, when the thread pool is unable to create a
thread, it presses on with the threads it has rather than raise.
Subsequent attempts to add work to the thread pool will cause the
pool to attempt to create the thread, as long as enough time has
passed since the most recent thread-creation failure.

Before this change, the thread pool wouldn't handle a
Thread.create exception, and the exception would get raised to
whatever code happened to be calling the Thread_pool function that
tried to create a thread, e.g. Thread_pool.add_work. This caused
In_thread.run to unexpectedly raise, and in turn
In_thread.syscall to unexpectedly raise, leading to:

"Fd.syscall_in_thread bug -- should be impossible"

Also, changed should be impossible text to please report, since
there may be other lurking rare exceptions that In_thread.syscall
can raise, and we'd like to hear about them.

We rely on thread-pool-stuck detection to report problems where the
inability to create threads causes the inability of the thread pool
to make progress. A tweak was needed to make that work -- now the
thread-pool-stuck warning is based on whether the thread pool has
unstarted work, rather than on whether the thread pool has an
"available thread". The latter would no longer work, since it is
now possible for the thread pool to have unstarted work and to
appear to have an available thread, i.e. num_threads <
max_num_threads.

core

Change some Bigstring functions to retry on EINTR rather than
raise.

The following functions (and their unsafe versions) were affected:

read

really_read

really_recv

really_write

really_send_no_sigpipe

Some other Bigstring functions, like input and output, already
retried on EINTR, so this change has precedent.

All of the affected stubs raise Bigstring.IOError on failure,
rather than Unix_error, which means the normal method for retrying
on EINTR doesn't work. In particular Async.Reader didn't retry
them, even though it was supposed to.

Additionally, the documentation for the following functions was
corrected to say that they raise =Unix_error= rather than =IOError=:

This required adding some additional type trickery to
Core_set_intf to indicate that the comparator for a given module
may or may not be fixed.

Added an optional iter parameter to Container.Make.

A direct implementation of iter is often more efficient than
defining iter in terms of fold, and in these cases, the results
of Container.Make that are defined in terms of iter will be more
efficient also.

jenga

Fix a hang.

Jenga could reach a state with a non-zero todo-count, but have no
jobs actually running, and then hang in this state forever. The hang
would be evident from a progress line with not all targets built and
with j=0+0 such as:

The new implementation requires 3 words per element, down from the 7
words required by the old implementation.

The new implementation pools elements so that they can be reused, so
there is no allocation in steady-state use.

The new implementation has dequeue_exn rather than dequeue, so
that one can dequeue without allocating 2 words.

Eliminated create'. One should just use create and explicit calls
to enqueue and dequeue_exn.

Eliminated dequeue_until_empty. One should use an explicit while
loop guarded by length and using dequeue_exn.

Moved Thread_safe_queue from Core_kernel to Core, since it's
thread related.

All in, there is now no allocation in a steady-state usage of
enqueueing and dequeueing elements, as opposed to 9 words per
enqueue+dequeue in the old implementation. This reduces the cost
from enqueue+dequeue taking 166-216ns to enqueue+dequeue_exn
taking 48-82ns (plus eliminating gc impacts). Here are some BENCH
results, the first table being the old implementation, and the
second table the new.

Name

Time/Run

mWd/Run

mjWd/Run

[thread_safe_queue.ml] enqueue + dequeue of immediate

183.89ns

9.00w

7.02w

[thread_safe_queue.ml] enqueue + dequeue of young object

216.69ns

11.00w

9.01w

[thread_safe_queue.ml] enqueue + dequeue_exn of old object

166.75ns

9.00w

7.02w

Name

Time/Run

mWd/Run

[thread_safe_queue.ml] enqueue + dequeue_exn of immediate

48.20ns

[thread_safe_queue.ml] enqueue + dequeue_exn of young object

81.96ns

2.00w

[thread_safe_queue.ml] enqueue + dequeue_exn of old object

48.30ns

Changed {Bigstring,Iobuf}.recvmmsg_assume_fd_is_nonblocking, when
no message is available, to return a negative number rather than
raise.

This was done for performance reasons, because raising an exception
is expensive, due to the stashing of the backtrace and the string
creation.

Added Iobuf.unsafe_resize.

Changed Bigstring.blit so that it doesn't release the OCaml lock
on map_file bigstrings.

The old behavior of releasing the lock for blits of (small)
bigstrings involving mmapped files was problematic and inconsistent.
Its cost is high, and fundamentally any access to a mapped bigstring
could cause some level of blocking.

Added time-related Arg_type.t values to Command.Spec.

Added module Type_immediacy, which has witnesses that express
whether a type's values are always, sometimes, or never immediate.

This code used to be in the Typerep_immediate library in typerep.

core_kernel

Added inline benchmarks for Array

Here are some of the results from the new benchmarks, with some
indexed tests dropped.

Changed the internal representation of Log.Rotation.t, but
t_of_sexp is backwards compatible, so existing config files will
continue to work.

Changed Udp.bind_any to use Socket.bind ~reuseaddr:false, to
ensure a unique port.

Added Tcp.Server.listening_socket, which returns the socket the
server is listening on.

Changed Tcp.Server so that if the listening socket is closed, the
server is closed.

Added to Udp.Config.t a max_ready : int field to prevent UDP
receive loops from starving other async jobs.

Improved File_tail to cut the number of fstat calls in half.

File_tail uses a stat loop to monitor a file and continue reading
it as it grows. We had made two fstat invocations per loop
iteration, using Async.Std.Unix.with_file which constructs an
Fd.t and therefore does it own fstat. Switching to
Core.Std.Unix.with_file with In_thread.run eliminated the extra
fstat.

async_unix

Improved Socket.accept to abort and return `Socket_closed when
the file descriptor underlying the socket is closed.

Added to Socket.bind a ?reuseaddr:bool argument, preserving the
default as true.

Added Fd.close_started, which becomes determined when close is
called on an Fd.t.

bin_prot

Fixed build on ARM.

core

Added Unix.Syslog module.

Changed Command.run to no longer ignore the first element of its
~argv parameter.

Made Time.Span.to_short_string show microsecond precision.

core_extended

Added Set_lang, a DSL for sets with constants, union, intersection,
and difference.

In Process, use epoll rather than select when possible,

This prevents errors when selecting on file descriptors numbered
greater than FD_SETSIZE (1024).

Removed Syslog module. There is now Unix.Syslog in core instead;
the APIs are not compatible, but they are similar.

core_kernel

Added Error.to_string_hum_deprecated that is the same as
Error.to_string_hum pre 109.61.

This fixed undesirable sexp escaping introduced in 109.61 and
restores the pre-109.61 behavior for the special case of
Error.of_string. A consequence of the removal of the custom
to_string_hum converter in 109.61 was that:

core_kernel

Renamed transfer, which was constant time, as blit_transfer,
which is linear time.

Removed partial_iter. One can use with_return.

Added singleton, filter, get, set.

For Error and Info, changed to_string_hum to use sexp_of_t
and Sexp.to_string_hum, rather than a custom string format.

Changed the output format of Validate.errors to be a sexp.

Added Hashtbl.of_alist_or_error and Map.of_alist_or_error.

Added String_id.Make functor, which includes a module name for
better error messages.

Exposed Bucket.size.

Changed the default for Debug.should_print_backtrace to be false
rather than true.

Usually the backtraces are noise.

Removed the tuning of gc parameters built in to Core, so that the
default is now the stock OCaml settings.

Such tuning doesn't belong in Core, but rather done per application.
Also, the Core settings had fallen way out of date, and not kept up
with changes in the OCaml runtime settings. We have one example
(lwt on async) where the Core settings significantly slowed down a
program.

Added Exn.raise_without_backtrace, to raise without building a
backtrace.

raise_without_backtrace never builds a backtrace, even when
Backtrace.am_recording ().

Made with_return faster by using Exn.raise_without_backtrace.

Improved with_return to detect usage of a return after its
creating with_return has returned.

ocaml_plugin

Added cmi's so that plugins can use lazy, recursive modules, and
objects.

pa_test

For <:test_result< >>, renamed ~expected to ~expect

sexplib

Added with sexp support for mutually recursive types with common
fields.

For instance:

typea = { fl : float } andb = { fl : int } withsexp

Closes #3

Fixed the behavior of sexplib on private types.

sexplib used to ignore the private modifier, which means generated
functions had the wrong type. Now, it generates a function with the
right type for sexp_of and refuses to generate anything for
of_sexp.

Fixed Macro's handling of nested :include's which was broken
with respect to paths.

Prior to this fix, :include's were broken with respect to the path
used to resolve the filename. Including a file outside the current
directory which included another file relative to that one would
break.

109.60.00

async_extra

async_kernel

Changed the scheduler to clear a job from its queue when it runs the
job, eliminating a performance regression from 109.57.

Clearing avoids spurious promotion of what would otherwise be dead
data associated with already-executed jobs.

async_unix

Fixed a bug in detection of the thread pool being stuck that could
overstate the amount of time the pool was stuck.

It had been incorrectly reporting the duration of the thread pool
being stuck if the pool had no work in it and then got enough jobs
to be stuck. In that situation, the duration included the time span
before the pool got stuck. If the pool had been idle long enough,
this could even spuriously abort the program.

comparelib

Fixed a type error in with compare of polymorphic variant inclusions.

core

Added Iobuf.unsafe_advance.

This can be used to benchmark inner loops that have redundant bounds
checking, to see if the checks matter. For example, see the
following two advance calls:

Fixed Lock_file.create to behave correctly if the target mountpoint
is out of space.

Previously in this situation, Lock_file.create would create an
empty lock and exit with exception trying to write pid/message
there. Subsequent runs would not able to read pid out of empty pid
file and blocking_create would block instead of removing defective
lock.

core_kernel

Added Gc.keep_alive, which ensures its argument is live at the point
of the call.

Added Sexp.With_text module, which keeps a value and the a sexp it
was generated from, preserving the original formatting.

custom_printf

Compatibility with warning 7 (method override)

faillib

Compatibility with warning 7 (method override)

pipebang

Compatibility with warning 7 (method override)

sexplib

Moved unix-specific code to a new object section, sexplib_unix

type_conv

Compatibility with warning 7 (method override)

109.58.00

async_extra

Changed Cpu_usage to use Core.Percent instead of float where
appropriate.

Made Bus.unsubscribe check that the subscriber is subscribed to
the given bus.

Made Log.t support with sexp_of.

Fixed Tcp.on_port 0 to return the port actually being listened on,
like Tcp.on_port_chosen_by_os.

Previously, a serverlistening on Tcp.on_port 0 would have its
Tcp.Server.listening_on as 0, which of course is not the port
the server is listening on.

async_kernel

Renamed the Async_core library as Async_kernel, to parallel
Core_kernel.

Someday Async_core will depend only on Core_kernel, but not yet.

Added a thread-safe queue of "external actions" that is checked
after each job.

Fixed a race condition in Clock.Event.abort.

Here is the race condition:

Clock.Event.at adds an alarm, its value is a job (let's call it
job1) with this run function:

letfire()=
t :=Happened;
Ivar.fill ready `Happened;

later a job (let's call it job2) aborting the clock event is
queued in the async scheduler

in the same cycle, the Timing_wheel.advance_clock fires the
alarm and job1 scheduled

at this point:

job1 and job2 are still pending

the alarm was removed so it is invalid

the clock event is still in the state Waiting

job2 is executed before job1: the clock event is still in the
Waiting state, so the abort tries to remove the alarm from the
timing wheel: CRASH

The bugfix is for Clock.Event.abort to check if the alarm has
already been removed from the timing wheel and if so, don't remove
it again.

Changed Monitor.try_with when run with ~rest:`Ignore, the
default, so that the created monitor is detached from the monitor
tree.

The detached monitor has no parent, rather than being a child of the
current monitor. This will eliminate recently observed space leaks
in Sequencer_table and Throttle, like:

Changed Async's scheduler to pool jobs rather than heap allocate
them, decreasing the cost of a job by 30-40%.

Changed the main scheduler queue of jobs to be an Obj_array.t that
is essentially a specialized Flat_queue (the specialization was
necessary for speed).

Also, cleaned up the scheduler run-job loop.

With these changes, the cost of a simple job decreases significantly
(30-40%), across a range of live data sizes. Here are the
nanoseconds-per-job numbers for a microbenchmark with the old and
new approaches.

Removed debug_space_leaks from Async's internals. It hadn't been
used in years.

async_unix

Improved fairness of the async scheduler with respect to external
threads, including I/O done in external threads.

The change is to add a thread-safe queue of "external actions" that
is checked after each job.

Previously, when a job given to In_thread.run finished,
In_thread.run would take the async lock, fill the result ivar and
run a cycle. The problem is that in some situations, due to poor OS
scheduling, the helper thread never had a chance to grab the lock.
Now, In_thread.run tries to take the lock:

if it can it does as before

if it can't it enqueues a thunk in the external actions queue and
wakes up the scheduler

With this change, the helper thread doing an In_thread.run will
always quickly finish once the work is done, and the async scheduler
will fill in the result ivar as soon as the current job finishes.

Fixed Epoll_file_descr_watcher.invariant to deal with the timerfd,
which has the edge-triggered flag set.

Added Writer.write_gen, a generic functor for blitting directly to
a writer's buffer.

core

Part of the Gc module used to be in Core because it used
threads. But it doesn't use threads anymore, so can be all in
Core_kernel.

Improved Iobuf support for copying to/from strings and bigstrings.

The new modules are Iobuf.{Consume,Peek}.To_{bigstring,string}.
They match a Blit-like interface. We don't yet implement the
Blit interface in all applicable contexts, but do use Blit.Make
and expose some of the operations we want in the forms we expect
them to take under a Blit interface.

Added Linux_ext.Timerfd.to_file_descr.

Added to Time.next_multiple an optional argument to control
whether the inequality with after is strict.

Added Time.Zone.local, a lazily cached Time.Zone.machine_zone ().

This is the first stage in a plan to make explicit timezones more
pervasive. First, they are made more convenient, by replacing the
relatively wordy Time.Zone.machine_zone () with Time.Zone.local.
This involves making the underlying timezone type lazy.

The next stage will be to remove machine_zone and use
Time.Zone.local everywhere instead. Then (it is hoped) instead of
of_local_whatever, we just say e.g. of_date_ofday
Time.Zone.local and currently-implicitly-local functions will be
able to switch over to explicit timezones without becoming too
verbose.

Added Timing_wheel.Alarm.null.

Made Unix.File_descr.t have with sexp.

Closes janestreet/async_unix#3

Fixed OpenBSD compilation failures in C stubs.

Fixed Lock_file.is_locked to require read permission, not write
permission, on the lock file.

pa_bench

Added support for inline benchmarks using the BENCH syntax, similar
to TEST.

This feature allows users to specify inline benchmarks in any library.

One can specify a benchmark using the following syntax:

BENCH"name"= expr

In the above, the value of expr is ignored. This creates
a benchmark for expr, that is run using the
inline_benchmarks_runner script from the command-line. This
workflow is similar to that of inline unit tests.

One can specify benchmarks that require some initialization using
BENCH_FUN. For example:

BENCH_FUN"name"=let t = create ()in
(fun() -> test_something t)

The function returned on the RHS of BENCH_FUN should have type unit
-> unit. One can specify benchmarks that have a variable parameter
using BENCH_INDEXED. For example:

The above snippet measures the time taken to create arrays of
different length. Indexed tests are useful in highlighting
non-linearities in the execution time of functions.

We can group benchmarks together into modules and the output of
inline_benchmarks_runner will reflect this grouping.

BENCH_MODULE"Blit tests"=struct
..some benchmarks..
end

For examples of all of the above see core_gc.ml and core_array.ml.

Only the generated inline_benchmarks_runner.exe depends on
Core_bench and other libraries. The library that includes the the
benchmarks itself does not have a dependency on Core_bench. Doing
this is important so that we can add benchmarks to Core and still
avoid cyclic dependencies. Finally, it is important to note that
adding inline benchmarks should have little effect on the execution or
module initialization time.

109.53.00

async

Bump version number

async_extra

Added module Bus, which is an intraprocess "broadcast"
communication mechanism.

Added Tcp.to_inet_address and to_unix_address.

Added Tcp.to_socket which creates a Tcp.where_to_connect from a
Socket.Address.Inet.t.

Module Weak_hashtbl is now implemented as a wrapper around
Core.Weak_hashtbl.

No intended change in behavior.

async_unix

Changed the scheduler to calibrate Time_stamp_counter every
second.

Improved error messages in the scheduler when epoll functions
raise.

Made Scheduler.reset_in_forked_process close the epoll file
descriptor.

bin_prot

Bump version number

core

Added Linux_ext.Epoll.close.

Added Weak_hashtbl module, moved from Async.

It had only been in Async to use Async's finalizers. The move
to Core exposes a bit more with respect to finalization so that
one can still implement Async.Weak_hashtbl, as well as do other
things (e.g. use Weak_hashtbl in Incremental, which does not use
Async).

Simplified the implementation of Weak_hashtbl to eliminate "entry
ids". They were only used to avoid removing a table entry that was
in use. But there is a more direct way to test for that -- just
check if the weak is None or Some.

Added an autoload file for utop

Disabled warning 40 in corebuild

core_bench

Fixed a bug in Core_bench where the linear regression was
sometimes supplied with spurious data.

This showed up when doing custom regressions that allow for a non-zero
y-intercept.

core_extended

core_kernel

Added Float.to_string_round_trippable, which produces a string
that loses no precision but (usually) uses as few digits as
possible.

This can eliminate noise at the end (e.g. 3.14 not
3.1400000000000001243).

Benchmarks:

New sexp:

Name

Time/Run

mWd/Run

Percentage

new Float.sexp_of 3.14

463.28ns

6.00w

48.88%

new Float.sexp_of e

947.71ns

12.00w

100.00%

Old sexp:

Name

Time/Run

mWd/Run

Percentage

old Float.sexp_of 3.14

841.99ns

178.00w

98.03%

old Float.sexp_of e

858.94ns

178.00w

100.00%

Much of the speedup in the 3.14 case comes from the fact that
format_float "%.15g" is much faster than sprintf "%.15g". And
of course the above does not capture any of the benefits of dealing
with shorter strings down the road.

Here are some detailed benchmarks of the various bits and pieces of
what's going on here:

Name

Time/Run

mWd/Run

Percentage

format_float '%.15g' 3.14

335.96ns

2.00w

32.71%

format_float '%.17g' 3.14

394.18ns

4.00w

38.38%

format_float '%.20g' 3.14

459.79ns

4.00w

44.77%

format_float '%.40g' 3.14

638.06ns

7.00w

62.13%

sprintf '%.15g' 3.14

723.71ns

165.00w

70.47%

sprintf '%.17g' 3.14

803.44ns

173.00w

78.23%

sprintf '%.20g' 3.14

920.78ns

176.00w

89.66%

sprintf '%.40g' 3.14

990.09ns

187.00w

96.41%

format_float '%.15g' e

357.59ns

4.00w

34.82%

format_float '%.17g' e

372.16ns

4.00w

36.24%

format_float '%.20g' e

434.59ns

4.00w

42.32%

format_float '%.40g' e

592.78ns

7.00w

57.72%

sprintf '%.15g' e

742.12ns

173.00w

72.26%

sprintf '%.17g' e

747.92ns

173.00w

72.83%

sprintf '%.20g' e

836.30ns

176.00w

81.43%

sprintf '%.40g' e

1_026.96ns

187.00w

100.00%

valid_float_lexem 12345678901234567

76.29ns

9.00w

7.43%

valid_float_lexem 3.14

9.28ns

5.00w

0.90%

float_of_string 3.14

130.19ns

2.00w

12.68%

float_of_string 1234567890123456.7

184.33ns

2.00w

17.95%

to_string 3.14

316.47ns

7.00w

30.82%

to_string_round_trippable 3.14

466.02ns

9.00w

45.38%

to_string e

315.41ns

7.00w

30.71%

to_string_round_trippable e

949.12ns

15.00w

92.42%

Replaced Float.min_positive_value with min_positive_normal_value
and min_positive_subnormal_value.

Added some functions to Float.O: abs, of_float, and
Robustly_comparable.S.

Small improvements to the Heap module.

Implemented Heap.iter directly rather than in terms of fold.

In heap.ml, fixed the idiom for using Container.Make.

Added an Int.O and other Int*.O modules, with arithmetic
operators, infix comparators, and a few useful arithmetic values.

Added Int.( ~- ), for unary negation.

Added Pool.unsafe_free.

Added Percent module.

core_profiler

Fixed a bug in Core_bench where the linear regression was
sometimes supplied with spurious data.

This showed up when doing custom regressions that allow for a non-zero
y-intercept.

ocaml_plugin

Bump version number

pa_ounit

Bump version number

pa_test

Bump version number

patdiff

Bump version number

re2

Bump version number

sexplib

Changed sexp_of_float to (usually) use as few digits as possible,
without losing precision.

Split the part of Sexplib that depends on Num into a separate
library, Sexplib_num.

This was done to eliminate the dependence of Core_kernel on Num,
which is not usable on javascript.

textutils

type_conv

109.52.00

async_extra

Fixed Sequencer_table.enqueue so that there is no deferred between
finding the state and calling the user function.

async_kernel

Changed Pipe.iter_without_pushback to never call f after
Pipe.close_read has been called.

The new behavior is like Pipe.iter.

Changed the implementation of Pipe.fold_gen and Pipe.transfer_gen
to be analogous to Pipe.iter_without_pushback, and in particular to
process as many elements as possible before calling values_available.

async_unix

Fixed a bug in Unix.mkstemp, which had a race because it used
Fd.create in a thread.

This bug affected Writer.with_file_atomic, save, save_lines,
and save_sexp, and could cause corruption of the async scheduler
data structures.

Changed async to never do set_nonblock on std{in,out,err}, which
allows Core I/O to use the standard file descriptors
simultaneously with async.

Before this change, the Core I/O libraries could (and sometimes did)
fail due to Sys_blocked_io.

Changed Pipe.iter_without_pushback to never call f after
Pipe.close_read has been called.

The new behavior is like Pipe.iter.

Changed the implementation of Pipe.fold_gen and Pipe.transfer_gen
to be analogous to Pipe.iter_without_pushback, and in particular to
process as many elements as possible before calling values_available.

Added ?expand_macros:bool argument to Reader.load_sexp*
functions, to support the new Sexplib macros.

Added an optional argument to Process.run to accept nonzero exits
as successful runs.

core

Added Unix.File_descr.equal.

Added Lock_file.Nfs.unlock, the Or_error version of
unlock_exn.

Improved the detail of the error messages exposed by
Lock_file.Nfs.create{,_exn}.

Added Unix.set_mcast_ifname, to control the interface used for UDP
multicast traffic.

Unfortunately, these aren't available in Core.Std.Debug, but only
in Core.Debug. That will be fixed in 109.49.

Made Time_stamp_counter compile on non x86-64 platforms.

Made Core.Std.Debug be Core.Debug rather than
Core_kernel.Debug.

This exposes the Debug.am* functions added in 109.48.

core_bench

Exposed an extensible form of make_command so that
inline-benchmarking and the other tools can add more commandline
flags.

A significant rewrite of Core_bench.

The rewrite provides largely the same functionality as the older
version. The most visible external change is that the API makes it
clear that Core_bench performs linear regressions to come up with
its numbers. Further, it allows running user-specified multivariate
regressions in addition to the built in ones.

The underlying code has been cleaned up in many ways, some of which
are aimed at improving the implementation of inline benchmarking
(the BENCH syntax, which has not yet been released).

core_extended

Removed Sexp.load_sexp_with_includes; one should use the new
Sexplib.Macro functions.

Added Blang-like languages Flang and Olang.

Flang -- terms over a field.

Olang -- predicates over an ordered set.

core_kernel

Added to Binary_packing module functions for packing and unpacking
signed 64-bit ints in little- and big-endian.

Changed the Comparator interfaces to no longer have with bin_io
or with sexp.

This was done since we almost always want a specialized map, and
we kept making the mistake of not overriding the generic one in the
three places needed.

Added Monad.map_via_bind, which one can use to create a standard
map function using bind and return.

Removed unnecessary signature constraints on the result of applying
Monad.Make.

Some time ago, Monad.Make changed from returning:

Swithtype 'a t ` 'a M.t

to returning:

Swithtype 'a t :` 'a M.t

so we no longer need to constrain the result of Monad.Make at its
uses to remove t.

Changed String.exists and String.for_all to iterate by
increasing index rather than decreasing.

Added with compare to module Ref.

Made Flags be Comparable, with the order consistent with bitwise
subset.

Cleaned up the implementation of Union_find.

Improvemed the code in union_find.ml:

Removed an assert false.

do not reallocate a parent node during compress. This should
result in more stability for sets memory wise.

Added implementation notes.

Renamed internal variant constructors.

Added unit tests.

Added Float.O, a sub-module intended to be used with local opens.

The idea is to be able to write expressions like:

Float.O.((3.+4.) >6./2.)

This idiom is expected to be extended to other modules as well.

Added a sexp_of_t converter to Type_equal.Id.

Replaced Univ.Constr with Type_equal.Id.

Added Debug.eprintf, analogous to eprint and eprints.

core_profiler

Exposed an extensible form of make_command so that
inline-benchmarking and the other tools can add more commandline
flags.

A significant rewrite of Core_bench.

The rewrite provides largely the same functionality as the older
version. The most visible external change is that the API makes it
clear that Core_bench performs linear regressions to come up with
its numbers. Further, it allows running user-specified multivariate
regressions in addition to the built in ones.

The underlying code has been cleaned up in many ways, some of which
are aimed at improving the implementation of inline benchmarking
(the BENCH syntax, which has not yet been released).

pa_ounit

Added -stop-on-error flag to inline_test_runner, to stop running
tests at the first failure.

This is useful if the remaining tests are likely to fail too or just
long to run.

pa_test

Added an optional ?message argument to <:test_eq<>>,
<:test_result<>> and <:test_pred<>>.

re2

Fixed bugs in Re2.Regexp.find_all and Re2.Regexp.find_first.

memory leaks on errors

unlikely garbage in their return values, or even segfaults
(especially unlikely for find_first)

sexplib

Added a Macro module, with load_sexp* functions that support
file includes and templates.

bin_prot

core

This module provides the fast function Time_stamp_counter.now ()
which is our best effort high-performance cycle counter for a given
platform. For x86 systems this retrieves the CPU's internal time
stamp counter using the RDTSC instruction. For systems that do not
have a RDTSC instruction, we fallback to using
clock_gettime(CLOCK_MONOTONIC).

Here is a benchmark of execution time in nanos and allocations in words:

Historically, the rate of increment of the TSC (sometimes referred to
as the TSC frequency) varied based of CPU overclocking, temperature,
load etc. On modern Intel CPU's the TSC is expected to be stable. On
Linux systems, the "constant_tsc" in /proc/cpuinfo indicates that the
machine has a stable TSC rate. While this module assumes that the TSC
is relatively stable, it can adapt to small variations in the TSC
frequency.

Changed Daemon.daemonize and daemonize_wait to leave the umask alone by default.

Previously, these had alwasy set the umask to 0, which means that
all app-harness programs and all binaries run from grass were creating
world-writeable (0666) files by default.

core_kernel

Added Error.to_info and of_info.

Significantly sped up Float.iround_* functions.

For iround_down_exn, the new version appears to use about 25% of the
CPU time of the old version on non-negative floats. For negative
floats it uses around 60% of the CPU time.

Name

Time (ns)

% of max

old iround_down_exn pos

15.02

95.23

new iround_down_exn pos

3.75

23.75

old iround_down_exn neg

15.78

100.00

new iround_down_exn neg

9.80

62.10

Added Binary_searchable.Make functor to core, and used it in Array and Dequeue.

Fixed Bounded_int_table to match Invariant.S2.

Added to Pool support for 10-, 11-, and 12- tuples.

Added functions to the Gc module to get usage information without allocating.

async_kernel

async_unix

Added Fd.every_ready_toFd.interruptible_every_ready_to which
register a callback to be called every time the fd becomes ready.

These can significantly reduce allocation.

Renamed Fd.ready_to_interruptible as Fd.interruptible_ready_to.

Changed Fd.ready_fold to use Fd.interruptible_ready_to, to
improve its performance.

core

Added Core.Std.phys_same, which is like phys_equal, except has a
more general type.

valphys_equal : 'a -> 'a -> boolvalphys_same : _ -> _ -> bool

phys_same is useful when dealing with existential types, and one
has a packed value and an unpacked value that one wants to check are
physically equal. One can't use phys_equal in such a situation
because the types are different.

Added Iobuf.set_bounds_and_buffer and set_bounds_and_buffer_sub,
which make it easier to use with zero allocation.

core_kernel

Implemented Dequeue.iter directly, instead of as a specialization
of fold.

Extended random tests to cover iter.

109.42.00

async_extra

Fixed Log.Blocking so that when async is running it writes the message in syslog before failing with an exception.

async_kernel

In ASYNC_CONFIG, replaced alarm_precision and timing_wheel_level_bits with timing_wheel_config.

This parallels the new Timing_wheel.Config module.

async_unix

Added Reader.drain.

valdrain : t -> unitDeferred.t

Added Writer.with_close.

valwith_close : t -> f:(unit-> 'aDeferred.t) -> 'aDeferred.t

bin_prot

Backed out the changes introduced in 109.41

core

Removed Zone from Core.Std; use Time.Zone instead.

Removed Time.Date; use Date instead.

Improved the performance of Piecewise_linear by using arrays with binary search on indices.

The previous implementation Piecewise_linear used (float * float)
list (a list of (x,y) pairs) to represent piecewise linear functions,
with a linear search through the knots to evaluate the function at a
point. This type is now:

{ x : floatarray;y : floatarray
}

and the implementation uses binary search to identify the correct array index.

Here are the costs of the get function under the old (list) and new (array)
implementations for various numbers of knots:

core_bench

Columns that have a + prefix are now always displayed, whereas
columns that don't are displayed only if they have meaningful data.

Added the ability to reload saved metrics (benchmark test data) so
that bench can re-analyze them.

core_kernel

Added Map.of_alist_reduce.

This function is a natural addition alongside of_alist_fold. Its
advantage is that it does not require an init argument like
of_alist_fold. Moreover, it does not involve option types, like
List.reduce does in order to handle the empty list case.

core_profiler

Columns that have a + prefix are now always displayed, whereas
columns that don't are displayed only if they have meaningful data.

Added the ability to reload saved metrics (benchmark test data) so
that bench can re-analyze them.

ocaml_plugin

Added option -strict-sequence, which is set to true by default.

type_conv

Fixed the generated code of typerep and sexplib on sum types containing True or False.

Without this fix, typerep would wrong constructor names for
Blang.t, for instance.

Variantslib has the same problem but applying the same fix there
would still not make the generated code compile because the generated
code would contain labels and variable named true or false.

Other syntax extensions should not be affected because they don't
build strings from constructor names.

core_bench

Added support for additional predictors like minor/major GCs and
compactions, using multi-variable linear regression.

Replaced linear regression with multi-variable linear regression.
The original algorithm estimated the cost of a function f by using
a linear regression of the time taken to run f vs the number of
runs. The new version adds the ability to include additional
predictors such as minor GCs, compactions etc.

This allows a more fine-grained split-up of the running costs of a
function, distinguishing between the time spent actually running f
and the time spent doing minor GCs, major GCs or compactions.

Added a forking option that allows benchmarks to be run in separate
processes.

This avoids any influence (e.g. polluting the cache, size of live
heap words) they might otherwise have on each other.

core_kernel

Implemented Heap.iter directly instead of in terms of fold.

core_profiler

Added support for additional predictors like minor/major GCs and
compactions, using multi-variable linear regression.

Replaced linear regression with multi-variable linear regression.
The original algorithm estimated the cost of a function f by using
a linear regression of the time taken to run f vs the number of
runs. The new version adds the ability to include additional
predictors such as minor GCs, compactions etc.

This allows a more fine-grained split-up of the running costs of a
function, distinguishing between the time spent actually running f
and the time spent doing minor GCs, major GCs or compactions.

Added a forking option that allows benchmarks to be run in separate
processes.

This avoids any influence (e.g. polluting the cache, size of live
heap words) they might otherwise have on each other.

109.38.00

async_extra

In Rpc, exposed accessors for binary protocol values.

For example, this allows one to write a wrapper for Pipe_rpc that
allows for the easy re cording and replaying of values the come over
the pipe.

async_unix

Added Reader.of_pipe and Writer.of_pipe, for converting from string Pipe.t's.

These can be used to add arbitrary transformations (e.g. encryption, compression)
to code that expects plain file- or socket-based readers and writers.

109.37.00

core

Command.run now calls Exn.handle_uncaught so you don't have to.

Fixes for building on FreeBSD.

Fixed Blang to build with OCaml 4.01.

In blang.mli:

Blang.t is a private variant type, Blang.Stable.V1.t is a private
variant type, and client code knows Blang.t = Blang.Stable.V1.t.
Previously, this was done in a strange way, using with type 'a t =
private 'a t on the signature of Blang.Stable.V1. In addition to
being strange, this line no longer builds in OCaml 4.01, which
caused problems for building Real World Ocaml.

This patch changed the code to something much more straightforward,
although not quite so straightforward as we expect to be able to
achieve once a nonrec bug is fixed.

core_kernel

Added Core.Std.Poly as a short name for
Core.Std.Polymorphic_compare.

Exposed module Core.Std.Decimal.

109.36.00

async_unix

Added Process.run_lines, which runs a process and returns stdout as a list of strings.

This fell out of sync with OCaml's hash function, and was providing worse hash values.

Fixed Obj_array.singleton to never create a float array.

Also made it clearer that Obj_array.copy could never create a float
array.

Changed Pool.create to allow zero-length pools.

Previously, Pool.create ~capacity:0 had raised, which made it easy
to write code that blows up on edge cases for no apparent reason. For
example, Heap.copy was written in a way that copying an empty heap
would blow up (regardless of its capacity), and Heap.of_array would
also blow up on an empty array.

Added String.split_lines.

(** [split_lines t] returns the list of lines that comprise [t]. The lines do not include the trailing ["\n"] or ["\r\n"]. *)valsplit_lines : t -> tlist

pa_ounit

Simplified so that it does not generate unnecessary top-level bindings.

It had been hiding quite a few unused import warnings.

textutils

In Ascii_table module, added support for displaying table using ASCII characters instead of Unicode.

This is motivated by the need to use Core_bench in contexts where
the extended ASCII character set is not suitable for displaying
tables.

such that these functions cause no allocations by themselves. The
assumption being that 63-bit ints should be large enough to express
total allocations for most programs. On 32-bit machines the numbers
may overflow and these functions are not as generally useful.

These functions were added because doing memory allocation debugging
with Gc.quick_stat as the primary means of understanding
allocations is difficult: tracking down allocations of the order of
a few hundred words in a hot loop by putting in lots of quick_stat
statements becomes too intrusive because of the memory allocations
they cause.

Here are some benchmarks of existing Gc functions and the newly
added functions:

that throws a nice exception if the values are not equal according to f.

<:test_pred< type >> expands to a function of type

?here:Lexing.position list -> (type -> bool) -> type -> unit

that throws a nice exception if the predicate fails on the value.

109.33.00

async_extra

Change Log.Global to by default send all output, including `Info,
to stderr.

Replaced Log.Output.screen with Log.Output.stderr. There is now
also and Log.Output.stdout.

109.32.00

async_extra

Added Dynamic_port_writer.

Dynamic_port_writer solves the problem of communicating a
dynamically selected tcp port from a child process to its parent.

async_kernel

Improved the batching of Pipe.fold and other Pipe functions that handle batches.

Previously, such functions used a loop with Pipe.read. This
didn't batch as well as it might. If values were put in the pipe
after the read becomes determined but before the values are
handled, then they wouldn't be handled until the next batch. Now,
batching functions use values_available and then pull elements out
of the pipe synchronously after waking up. This makes the batch as
large as possible.

core

In Iobuf, added duals to flip and snapshot to work on the high end of the window.

flip has been renamed to flip_lo. The dual of flip_lo is the
newly added flip_hi, and shifts the window forward to continue
reading, rather than back to switch from writing to reading, as
flip_lo does.

flip_hi, in practice, needs snapshots of the upper bound of the
window, we split Snapshot into Lo_bound and Hi_bound and
introduced bounded versions of flip_lo, compact, and flip_hi to
support buffers which are only partially filled, but have
substructure, like packet buffers.

This is in mlre2__create_re if we fail to compile the regular
expression. Notice how we delete the re object before we use the pts
to its' error string. (Notice that in C++ c_str() returns a pointer
to the internal data of the string object it does NOT create a copy
and error() just returns a reference to the regular expression objects
error string member *error_).

So if caml_raise_with_string has to allocate on the heap to create
the exception or the copy of the string that might invalidate the ptr
before we will copy it.

109.31.00

async_unix

Renamed Reader.read_one_chuck_at_a_time_until_eof as Reader.read_one_chunk_at_a_time,
and added a new case to the handle_chunk callback.

The name change is to reflect that one can stop early, before EOF.

The new handle_chunk case allows one to specify the number of bytes
that were read in the Stop case.

Also, changed read_one_chunk_at_a_time to use do_read instead of
just locking the file without unlocking it. This allows subsequent
read operations to read from the file.

core_kernel

Renamed some functions in module Lazy: dropped the lazy_ prefix from is_val, from_val, and from_fun.

ocaml_plugin

Fixed OCaml Plugin on CentOS 5 -- it had problems because the generated c files did not end with a newline.

Finished the transition from Command_deprecated to Command.

109.30.00

async_kernel

Added function Throttle.kill.

Throttle.kill aborts all jobs that have been enqueued but not
started, and immediately aborts all jobs subsequently enqueued.

Split out Throttle debugging and unit-testing code into their own
modules.

Changed the semantics of Throttle.enqueue on a dead throttle so that
the exception is sent to the monitor rather than raised synchronously.

This gives more uniform treatment to the race between enqueueing a
job and an already running job raising. Now the enqueued job is
always aborted, whether enqueued before or after the raise.

Added an ASYNC_CONFIG option to print debug messages when Monitor.send_exn is called.

This is useful when one is debugging an application in which an
exception is being unexpectedly swallowed.

Allow one to dynamically configure the behavior of =Monitor.try_with=.

This is to allow experimentation with different handling of
asynchronous exceptions after Monitor.try_with has become
determined.

async_unix

Changed aync's scheduler to use epoll rather than select by default.

This is based on a dynamic test to see whether timerfd_create works.

Added support for "busy polling".

This runs a thread that busy loops running user-supplied polling
functions. The busy-loop thread is distinct from async's scheduler
thread, but it acquires the async lock so that the user-supplied
function can do ordinary async operations, e.g. fill an ivar.

Busy polling is useful for a situation like a shared-memory ringbuffer
being used for IPC. One can poll the ringbuffer with a busy poller,
and then when data is detected, fill some ivar that causes async code
to handle the data.

Added Async.Fd.clear_nonblocking.

This clears the nonblocking bit on the file descriptor underlying the
fd, and causes async to treat the fd as though it doesn't support
nonblocking I/O.

This is useful for applications that want to share a file descriptor
between async and non-async code and want to avoid Sys_blocked_io
being seen by the non-async code.

core

Created submodule Core.Signal.Expert module.

This is for functions previously in Core.Signal that introduce
multithreading semantics, and are hence hard to reason about and
should only be used by experts.

core_bench

Report compaction stats

core_kernel

Added module, Core.Blit, which codifies the type, implementation, and unit-testing of blit functions.

Added remove_zero_flags option to Flags.Make, to support flags that are zero.

This fixes a problem with Flags.Make on CentOS 5 because O_CLOEXEC is 0 there.

Removed Pool.None, and folded Pool.Obj_array into Pool proper.

Pool.None had its day, but Pool.Obj_array dominates it, so we don't need it any more.

core_profiler

Report compaction stats

ocaml_plugin

Support for Mac OSX

Removed the dependency of ocaml-plugin on objcopy and /proc.

109.28.00

async_extra

Fixed an error message in Versioned_rpc that was swapping which
versions were supported by the caller and the callee.

async_kernel

Eliminated a messy dependency cycle in async_core, so that
Monitor.t no longer contains a Tail.t.

This messiness caused the need for the Raw signature, for the
various Scheduler_dependent functors, for making various types
polymorphic in execution_context, and then instantiating the
polymorphism later.

The cause of the problem was that Monitor contained a Tail. We
eliminated that, so that there is no longer a cycle, and defined
things in order:

Monitor
Execution_context
Job
Scheduler
Ivar
Deferred
Tail
Stream

Replaced the errors tail from the monitor type:

errors : (exn, 'execution_context) Raw_tail.t;

with a list of handlers:

mutable error_handlers : (exn -> unit) list;

Cleaned up all the messiness caused by the cycle -- eliminated the
Raw signature, the Scheduler_dependent functors, and the
unnecessary polymorphism.

Cleaned up the long standing annoyance with Async.Stream, in which
we couldn't expose the next sum type and people had to use
annoying Stream.of_raw calls. Stream.of_raw is now gone.

core

Moved Timing_wheel from Zero.

core_extended

In Shell functions, made the amount of captured stderr/stdout
configurable.

core_kernel

Moved all the contents of the Zero library into Core, mostly
into Core_kernel.

We want to start using Zero stuff more in Core, which couldn't
be done with Zero as a separate library.

Everything moved into Core_kernel, except for Timing_wheel,
which moved into Core proper, due to its dependence on Time.

Renamed Flat_tuple_array as Flat_array.

Added Dequeue.{front,back}_index_exn

These are more efficient than using {front,back}_index and then
Option.value_exn.

Exposed Core.String.unsafe_{get,set}.

pipebang

Fixed pa_pipebang in the toplevel.

pa_pipebang had registered an AST filter for implementations but
not for the toplevel.

type_conv

Fixed an issue with type_conv in the toplevel.

Used AST filters for the _no_unused_value_warning_ machinery.
type_conv modifies the parser but it didn't work well in the
toplevel.

Removed the parsing_mli reference, an instead always add the
special _no_unused_value_warning_ type and just strip it for
signature items.

109.27.00

async_extra

Added function Versioned_typed_tcp.Client.shutdown.

Added new module Sequencer_table, which is a table of
Throttle.Sequencer's indexed by keys.

async_kernel

async_unix

Fixed a performance problem in the scheduler due to repeated calls
of Timing_wheel.min_elt.

Timing_wheel.min_elt is an important part of async, since the
scheduler calls it once per cycle to know when to timeout for
epoll or select. This causes a problem if min_elt is slow and
called repeatedly, which happens in an application where the next
clock event is a second out, and yet there are lots of cycles per
second.

Timing_wheel.min_elt now caches the minimum element, which
eliminates the problem.

Fixed async's clock to work on 32-bit machines.

With the change to Timing_wheel in 109.22, async no longer worked
on 32-bit machines, due to the clock overflowing. This is because
it is initialized to Time.epoch, and can only handle 6 days.

The fix now in place is to start the clock at Time.now () rather
than Time.epoch.

Added many functions to Async.Sys so that it looks more like
Core.Sys.

Changed Reader.read_one_chunk_at_a_time_until_eof to not destroy
the reader buffer.

Destroying the buffer failed if user code held on to the buffer.

comparelib

Changed how with compare treats option's so that None < Some,
like Pervasives.compare.

Previously, with compare had treated Some < None.

core

Disabled use of recvmmssg, which isn't available on our CentOS 5
machines.

Defined Option.compare using with compare so that their
comparisons are consistent.

Cleaned up the Dequeue module's interface and implementation.

The interface now matches the conventions used elsewhere in Core.
The new implementation is also cleaner and more efficient.

Reimplemented the Stack module to improve performance, and renamed
the old implementation as Linked_stack.

Using an array rather than a linked list is a more efficient and
traditional implementation. Pushing to the stack now does not
require allocation, except in the rare case when the stack grows.

One downside is that Stack.clear is now O(n) rather than O(1).

This change also eliminates the Stack.Empty exception, so any code
matching on that exception should fail to compile, and should be
changed to depend on option-returning top and pop operations.

Improved Lock_file.Nfs.

Allowed an arbitrary message to be stored and retreived.

Fixed a case where create might throw an exception.

Delete both files used for locking when we unlock.

Split Core into Core_kernel and Core.

Core_kernel is composed of all modules of Core that do not
depend on unix or threads, and Core contains the rest and depends
on Core_kernel.

The goal is to have a very portable standard library that can
especially run on exotic environment such as Javascript.

So that code that directly refers to Core (rather than Core.Std)
for modules that have moved to Core_kernel, we included "proxy"
modules in Core that simply include the corresponding module from
Core_kernel.

Fixed Core.Flags to build on 32-bit machines.

It had contained a unit test with an integer literal too large to be
accepted by the compiler when building on a 32-bit machine.

core_bench

Added R^2 error estimation.

Adding this metric should give us a sense of how closely the given
values fit a line. Even dots that are fairly scattered can give
tight confidence intervals. We would like to have to number to have
a sense of how much noise we have.

core_profiler

Added R^2 error estimation.

Adding this metric should give us a sense of how closely the given
values fit a line. Even dots that are fairly scattered can give
tight confidence intervals. We would like to have to number to have
a sense of how much noise we have.

custom_printf

Added missing registration of custom_printf's Camlp4 filter so
that it works in the toplevel.

109.24.00

async_extra

Made the Caller_converts interface in Versioned_rpc use the
Connection_with_menu idea introduced in Both_convert.

async_kernel

Reworked the Throttle module.

Made both Throttle.t and Throttle.Sequencer.t instances of the
same type, using a phantom type to distinguish them. Removed all
Throttle.Sequencer functions -- one can now use the Throttle
functions directly.

Added new functions:

(*** [max_concurrent_jobs t] returns the maximum number of jobs that [t] will run concurrently. *)valmax_concurrent_jobs : (_, _) t_ -> int(*** [num_jobs_running t] returns the number of jobs that [t] is currently running. It is guaranteed that if [num_jobs_running t < max_concurrent_jobs t] then [num_jobs_waiting_to_start t = 0]. That is, the throttle always uses its maximum concurrency if possible. *)valnum_jobs_running : (_, _) t_ -> int(*** [num_jobs_waiting_to_start t] returns the number of jobs that have been [enqueue]d but have not yet started. *)valnum_jobs_waiting_to_start : (_ , _) t_ -> int(*** [capacity_available t] becomes determined the next time that [t] has fewer than [max_concurrent_jobs t] running, and hence an [enqueue]d job would start immediately. *)valcapacity_available : (_, _) t_ -> unitDeferred.t

Replaced the Pipe used inside a Throttle with a Queue, and
simplified the implementation. This fixed a bug in
num_jobs_waiting_to_start, which could have missed a job that was
not in the pipe but had not started.

async_unix

Changed Reader.close so that it frees the reader buffer using Bigstring.unsafe_destroy.

This is an improvement over the previous situation, in which the
buffer wasn't freed until its finalizer fired.

Fixed a bug in Reader.read_bin_prot.

It was missing a try-with that could cause it to raise without
cleaning up the reader.

core

Added module Core.Iobuf, a module aimed at zero-copy I/O.

An iobuf is a bigstring combined with a position and length, that
defines a contiguous region of bytes in the bigstring. Operations on
an iobuf operate relative to start of the region and cannot look
outside the region.

Added module Core.Validated for creating an abstract type that
ensures a validation function has been run.

Added function Bigstring.recvmmsg_assume_fd_is_nonblocking, which
allows one to read a number of UDP packets with a single system
call.

Fixed Unix.create_process on OSX.

109.23.00

core

Exposed Core.Std.Flags module.

Made the Heap module implement Container.S1.

Added module Ref.Permissioned, which is a ref with read_only /
read_write access control.

Exposed the unique id in Type_equal.Id.

This allows, e.g. hash tables indexed by unique ids.

Removed the use of Obj.magic from the implementation of
Type_equal.Id.same.

It is not needed because the Id.t contains a Uid.t and we can
just use Uid.equal.

109.21.00

async_unix

Added Unix.remove.

core

Massively improved the signatures of Map and Set, both for
readability and ocamldoc, as well as improved type error messages.

Added an optional argument to Command.run that can be used to
specify default flags from a user config file.

The optional argument can extend the command line based on the path
to the command.

Rename module Weekday as Day_of_week.

The name Weekday conflicted with ordinary usage of "weekday" to
mean Monday through Friday.

Changed sexp_of_t for {Month,Ofday,Time,Time.Span}.{Set,Map} to
use the nice sexp format of the underlying atomic type.

Previously, the converter had used thes raw type (float, int,
etc.). t_of_sexp still accepts both formats; we will remove the
ability to accept the raw format in the distant future.

This output-format change was planned when we originally in 108.06b
improved those t_of_sexp functions to accept both formats.

Added Unix.remove.

Removed some IFDEF's connected to OCaml <4 support.

109.20.00

async_kernel

Added the ability for a Throttle to have resources that are exclusively available to running jobs.

async_unix

Set close-on-exec for both ends of the pipe used to wake up the scheduler.

core

Wrapped Unix.wordexp in an Or_error.t since it is not available on all systems.

Added function Process_env.parse_ssh_client.
This gets the address from which you're currently ssh'd in.

Added to Unix module the ability to get and set IP_MULTICAST_LOOP and IP_MULTICAST_TTL.

Exposed module Core.Std.Ref, which was previously only available via Core.Ref.

Remove Mutex.am_holding_mutex and changed the type of Mutex.try_lock.

With NPTL it is impossible to determine which thread is holding the
lock. So, Mutex.am_holding_mutex is unimplementable. Also,
Mutex.try_lock was incorrect because it claimed to raise if one was
attempting to recursively lock. Since it's not possible to
distinguish between recursive locking and the lock being held by
another thread, we changed the type to make this clear:

valtry_lock : t -> [ `Already_held_by_me_or_other | `Acquired ]

Removed our custom version of the OCaml runtime's core_sys_open function.

There used to be a bug in the OCaml runtime, PR#5069, in which
open_{in,out}_gen could block while holding the OCaml lock, because
they made a call to fcntl outside the blocking section. We had our
own C code with the bug fix and re-exposed the fixed versions of the
functions in Core.

The bug in OCaml has been fixed, so we have removed our patched
function from Core.

In unix_stubs.c, switched from using FNM_FILE_NAME to FNM_PATHNAME.

The GNU project introduced FNM_FILE_NAME as a non-portable synonym for
FNM_PATHNAME.

We were using pre-processor macros to define FNM_FILE_NAME as
FNM_PATHNAME if unavailable, but it is simpler to just use the more
portable FNM_PATHNAME everywhere.

ocaml_plugin

Removed a test that (rarely) failed nondeterministically.

sexplib

Renamed converter generated by with sexp for polymorphic variants so it is hidden from the toplevel.

of_sexp created a value named <type>_of_sexp__ to handle
polymorphic variants. To hide it from the toplevel, we renamed it as
__<type>_of_sexp__. We kept the __ suffix to avoid any confusion
with a type named __<type>.

This returns the input list with consecutive duplicates removed, and
doesn't change the order of the remaining elements.

Added module User_and_group, which is a pair of a unix username
and primary unix group.

The string/sexp converters follow the usual unix convention of
<user>:<group>.

Added function Date.first_strictly_after : t -> on:Weekday.t -> t.

first_strictly_after t ~on:day_of_week returns the first
occurrence of day_of_week strictly after t.

Added functor Type_equal.Lift.

It is always safe to conclude that if type a equals b, then type
a X.t equals b X.t, for any type X.t. The OCaml type checker
uses this fact when it can. However, sometimes, e.g. when using
Type_equal.conv, one needs to explicitly use this fact to
construct an appropriate Type_equal.t. The Type_equal.Lift*
functors do this.

109.15.00

async_extra

Added new module Versioned_rpc.Both_convert to allow the caller
and callee to independently upgrade to a new rpc.

This is a new flavor of Versioned_rpc in which both sides do some
type coercions.

async_unix

The epoll-based scheduler now supports sub-millisecond timeouts,
using Linux_ext.Timerfd.

Async still uses the select-based scheduler by default. We plan
to switch the default to epoll in a few weeks, once we have done
more testing.

Eliminated module Work_group, which was for limiting the number of
threads used by jobs.

This was a little-used module that significantly complicated the
implementation of the Async thread pool.

One should consider using a Throttle instead.

Along the way, fixed a bug in Async helper threads in which the
finalizer could fire too early, causing an unhandled exception. The
fix involves relaxing the requirements on when
Thread_pool.finished_with_helper_thread functions can be called,
allowing it to be called while the helper thread still has work, but
so long as no future work will be added.

core

Changed the tolerance of Time.Robustly_compare functions from
1E-7 to 1E-6.

Fixed the names of some toplevel pretty-printers, which referred to
nonexistent modules.

Fix some of the pp's for Core which are used to install printers
in the top-level. Some of the toplevel printers refer to
non-existent modules like Core.Nativeint.pp; this feature changed
to the correct name, like Core.Std.Nativeint.pp.

Added to module Unix functionality for getting and setting flags
in the open-file-descriptor table.

Added a ?wakeup_scheduler:bool optional argument to functions in
the Thread_safe module.

The default is true, which continues the behavior that has been in place since 109.09.
However, once can use ~wakeup_scheduler:false to reduce CPU use, in return for increased
latency (because the scheduler won't run a cycle immediately).

core

Fixed major performance problem with hashing in Int.Table.

Our Int.Table.replace was 3 times slower than polymorphic hash
table and find was 8 times slower.

Added Bigstring.{pread,pwrite}, which allow reading and writing at
a specific file offset.

Added module Nothing, which is a type with no values.

This is useful when an interface requires you to specify a type that
you know will never be used in your implementation.

Changed Identifiable.Make so that it registers a pretty printer.

Identifiable.Make now uses Pretty_printer.Register. This
requires all calls to Identifiable.Make to supply a val
module_name : string.

Made Core.Zone match the Identifiable signature.

Made polymorphic equality always fail on Core.Map.t and
Core.Set.t.

Before this change, polymorphic equality on a Core.Map or a
Core.Set could either raise or return false. It returnd false
if the data structures were unequal, and raised if the data
structures were equal.

and polymorphic equality visits a block's fields in order. So, it
will detect unequal trees and return false, but if the trees are
equal, it will compare the comparators and raise because of the
functional value.

This change reversed the order of the fields so polymorphic equality
always fails.

custom_printf

initial import
Added support for %{<Module>} in printf-style format strings.

If you put ! before a format string, it allows the use of a spec
like %{<Module>} in the format string. For example, using
%{Time} wraps Time.to_string around the appropriate argument.

It also allows different formats for a given type:
%{<Module>.<identifier>} wraps <Module>.Format.<identifier>
around the appropriate argument. For example, %{Float.pretty}
would wrap Float.Format.pretty around the appropriate argument.

fieldslib

Made with fields expose first-class fields for private types while
preserving privacy.

There is now an additional phantom type in a first-class field that
prevents building or modifying elements of a private type.

One consequence of this change is that the Field.t type is now an
abstract type -- it used to be exposed as a record type. So, one
must, e.g., change field.Field.name to Field.name field.

109.13.00

async_kernel

Fixed Pipe.iter's handling of a closed pipe.

Fixed the handling by Pipe.iter and related foldy functions that
handle one element at a time, which behaved surprisingly with a pipe
whose read end has been closed. These functions had worked by
reading a queue as a batch and then applying the user function to
each queue element. But if the pipe's read end is closed during the
processing of one queue element, no subsequent element should be
processed. Prior to this fix, the iter didn't notice the pipe was
closed for read until it went to read the next batch.

Renamed Pipe.read_one as Pipe.read_one', and added
Pipe.read_one that reads a single element.

async_unix

Added Writer.write_line, which is Writer.write plus a newline at
the end.

This uses the linux-specifc socket option BINDTODEVICE to prevent
packets being received from any interface other than one named.

Fixed Unix.mkdir_p on Mac OS X.

109.12.00

async_extra

Made explicit the equivalence between type Async.Command.t and type Core.Command.t.

async_unix

Fixed a bug in Fd.syscall_in_thread.

The bug could cause:

Fd.syscall_in_thread bug -- should be impossible

The bug was that syscall_in_thread raised rather than returning Error.

Changed Tcp.connect and Tcp.with_connect to also supply the connected socket.

Supplying the connected socket makes it easy to call Socket
functions, e.g. to find out information about the connection with
Socket.get{peer,sock}name. This also gives information about the IP
address after DNS, which wouldn't otherwise be available.

One could reconstruct the socket by extracting the fd from the
writer, and then calling Socket.of_fd with the correct
Socket.Type. But that is both error prone and not discoverable.

core

Eliminated the notion of "preferred measure", so a Byte_units.t
is just a float.

Improved the performance of Array.of_list_rev.

The new implementation puts the list elements directly in the right
place in the resulting array, rather that putting them in order and
then reversing the array in place.

Benchmarking shows that the new implementation runs in 2/3 the time of
the old one.

Fixed Fqueue.t_of_sexp, which didn't work with sexp_of_t.

There was a custom sexp_of_t to abstract away the internal record
structure and make the sexp look like a list, but there wasn't a
custom t_of_sexp defined, so it didn't work.

Added Stable.V1 types for Host_and_port.

Removed Identifiable.Of_sexpable and Identifiable.Of_stringable,
in favor of Identifiable.Make

Identifiable.Of_sexpable encouraged a terrible implementation of
Identifiable.S. In particular, hash, compare, and bin_io were
all built by converting the type to a sexp, and then to a string.

Identifiable.Of_stringable wasn't as obviously bad as
Of_sexpable. But it still used the string as an intermediate,
which is often the wrong choice -- especially for compare and
bin_io, which can be generated by preprocessors.

Added Identifiable.Make as the replacement. It avoids using sexp
conversion for any of the other operations.

fieldslib

Added back Fields.fold to with fields for private types.

We had removed Fields.fold for private types, but this caused
some pain. So we're putting it back. At some point, we'll patch
with fields to prevent setting mutable fields on private types via
the fields provided by fold.

sexplib

A tiny lexer improvement in lexer.mll. Used
lexbuf.lex_{start|curr}_pos instead of
lexbuf.lex_{start|curr}_p.pos_cnum for computing the length of a
lexeme since the difference is the same.

109.11.00

async_extra

Exposed a version function in Pipe_rpc and State_rpc.

async_kernel

Extended Deferred.Or_error to parallel almost all of the
Core.Or_error interface.

Improved the performance of Clock.at, and added a more efficient
version, Clock.run_at.

Reworked the async heap of clock alarms to use async jobs as alarms.

Reworked Clock.at to use this and to not use abortable events,
which is a performance improvement.

Added a more efficient version of Clock.at, for the common
situation when one doesn't need a deferred.

(*** [run_at time ~f] is a more efficient version of [at time >>> f]. *)valrun_at : Time.t -> f:(unit-> unit) -> unit

async_unix

Added a check to fail if Scheduler.go is called more than once.

core

Added module Interned_string This has a functor for creating
modules of interned strings. It uses the very simple mechanism of
mapping distinct strings to consecutive ints.

Added value Hashtbl.find_and_remove.

fieldslib

with fields, for a type u that isn't named t, creates module
Fields_of_u rather than module Fields. This allows one to us
with fields on several types in the same structure.

109.10.00

async_extra

Fixed a race condition in Pipe_rpc and State_rpc. This race
could cause an exception to be raised on connection closing.

async_unix

Added Shutdown.do_not_finish_shutdown_before. This allows one to
add unit Deferred.t's that will delay the shutdown from
finishing. The implementation is more efficient than using
at_shutdown.

bin_prot

Improved error messages in presence of GADTs.

comparelib

Improved error messages in presence of GADTs.

core

Added |>, which means the same as |!, but is likely to replace
it someday. This is mostly because |> is an accepted notation
elsewhere, particularly in F#. In the future, we will consider
eliminating |! in favor of |>, so as to avoid the duplication.

Made module Lazy into a monad.

Renamed
List.stable_dedup_involving_an_application_of_the_set_functor as
List.stable_dedup_staged. Made it use Staged.t to make explicit
the expectation of partial application.

Added pretty printers for the toplevel to Error and Info.

fieldslib

Changed with fields on private types to not expose mutators or
creators.

pa_ounit

Rewrote pa_ounit to simplify execution order and work better with
functors.

Rewrote pa_ounit to solve its shortcomings with functors, namely
that functors need to be applied with TEST_MODULE for their tests
to be registered. The order of execution is also much simpler:
tests are executed inline, at the toplevel (or functor application
time). There is still a limitation: when a library doesn't have any
occurrence of TEST, TEST_UNIT, or TEST_MODULE inside of it,
the test runners are not set up, so tests inside of functors (from
other libraries) will not be executed. Running
inline_test_runner.exe is not going to run tests anymore; people
should run the inline_test_runner script instead. Backtraces are
now properly shown when exceptions are thrown.

sexplib

Improved error messages in presence of GADTs.

Made with sexp work with types containing as in signatures.

variantslib

Improved error messages in presence of GADTs.

109.09.00

async

Switched Async.Std's toplevel bindings for Deferred.Or_error's bind and map to use
Deferred.Result.
This allows them to be used with any 'error type, rather than just Error.t.

async_kernel

Fixed bug in Async.Throttle, in which jobs weren't started in order.

async_unix

Added module Thread_safe_pipe, for streaming data outside async into async.
This a more efficient and feature-ful way to send a sequence of values
from outside async into async than Thread_safe.pipe, which has been
eliminated.

Changed functions in Thread_safe to always wake up the scheduler.
Changed Thread_safe.run_in_async{,_exn} to not run a cycle, and
instead rely on the scheduler to run the cycle.

core

In Core.Std, exposed Or_error.ok_exn and Or_error.error

Removed some values exported by Core.Std.

Removed some values from Core.Std that weren't widely used, or we
didn't think should be exposed, including ascending, descending,
and equal, which use polymorphic comparison, and we want to
discourage.

Here's a guide to some of what was removed, and what one should now
use instead.

Disallowed <:sexp_of< with two underscores; using a single underscore instead.

Added Command.Spec.Arg_type.of_alist_exn as an alternative for of_map.
This captures the common pattern to create the map from an alist.

Improved the performance of Hashtbl.
Constrained hashtbl size to a power of two and used a bitmask rather
than mod operation for finding hash buckets.

Improved the performance of Univ, using the Type_equal GADT.
The new implementation improves the run-time and space usage over
the old one. In the old implementation, a Univ.t was represented
as record with three fields: an exception, a string, and a closure.
Creating a univ required allocating three heap blocks, the exception
(3 words), the closure (3 words), and the three-field record (4
words). In the new implementation, a Univ.t is represented as a
2-field heap block containing the Constr.t and the value.
Creating a univ allocates that single 3-word block, improving on the
10 words needed previously.

Matching on univs is also faster. In the old implementation,
matching on a univ required making a function call, testing
exception equality, and allocating a Some block. Now, it does
just the test and allocation. Furthermore, it is possible to use
does_match and match_exn to avoid the allocation.

sexplib

Fixed an unused rec warning in the code generated by pa_sexp in
rare cases.

109.08.00

async_extra

Added module Async.Command
This is Core.Command with additional async functions. In particular
it contains a function async_basic that is exactly the same as
Core.Command.basic, except that the function it wraps returns
unit Deferred.t, instead of unit. async_basic will also start the
async scheduler before the wrapped function is run, and will stop the
scheduler when the wrapped function returns.