Math.NET, distributed computing and how an electrical engineer sees the world of complex software

F# Interactive (FSI) is a very convenient environment to execute pieces of F# code on the fly. You can even reference managed assemblies using the #r and #I preprocessor directives. However, if one of the referenced assemblies tries to use a native DLL using p/invoke you might end up with a DllNotFoundException even if the native DLL is in the same folder as the managed assembly, and if the folder has been included with the #I directive. Note that it is not possible to reference native DLLs explicitly in .Net.

The reason is that finding and loading such DLLs in .Net works the same way as all native applications in Windows and follows the standard search order. When launched from within VisualStudio, the working directory of the F# Interactive process is the path where it is installed, in my case C:\Program Files (x86)\Microsoft SDKs\F#\3.0\Framework\v4.0. Naturally it has no chance to find a DLL in your script folder and fails.

There are multiple ways how you can tell Windows where to look for the DLL:

Math.NET Numerics is well on its way towards the next major release, v3.0. A first preview alpha has already been pushed to the NuGet gallery, even though there’s still a lot to do. If you’d like to understand a bit better where we currently are, where we’re heading to, and why, then read on.

Math.NET Numerics v2.6, released in July 2013, is focused on filling some gaps around the very basic numerical problems of fitting a curve to data and finding solutions of nonlinear equations. As usual you’ll find a full listing of all changes in the release notes. However, I’d like to take the chance to highlight some important changes, show some code samples and explain the reasoning behind the changes.

Most .Net libraries should also work on Linux or OS X thanks to Mono. But, do they really? How do you verify and test that, without installing Mono on your Windows development box, or setting up a separate Linux box? Or how to compile and test some local files without installing any .net or mono dev tools at all?

Vagrant comes in handy here. Vagrant may not be very well known among .Net developers on Windows yet, so let the Vagrant team introduce it in their own words: “Vagrant is a tool for building complete development environments. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases development/production parity, and makes the “works on my machine” excuse a relic of the past.” In my words, Vagrant lets you define a standardized development or test environment for your project that works exactly the same everywhere, no matter what OS you’re on or how you’ve set it up. It uses virtual machines in the background, but you neither see nor care about them much.

Math.NET Numerics v2.5, released in April 2013, is focused on statistics and linear algebra. As usual you’ll find a full listing of all changes in the release notes. However, I’d like to take the chance to highlight some important changes, show some code samples and explain the reasoning behind the changes.

Linear algebra is one of those areas where performance can be essential, but
also one where native optimizations can make a huge difference. That’s why in Math.NET Numerics
we implemented linear algebra on top of a provider abstraction where providers can be exchanged.

Out of the box Math.NET Numerics only includes a fully managed provider which is supported on
all platforms, but unfortunately is also rather slow. This doesn’t matter much for most problems,
but if you’re working with very large dense matrices it can be a deal breaker. That’s why we’ve
added some helper projects you can use to compile your own native provider, but that is still
quite involved and requires some experience around C or C++. Not any more, kudos to @marcuscuda!

Since Math.NET Numerics v2.4
we begin to distribute native providers as NuGet packages, starting with one based on Intel MKL.
Enabling native algorithms becomes almost as simple as adding a NuGet package to your project.

GitHub recently started publishing all pull request as special git refs.
This is awesome, since it makes it trivial to checkout out and work with them
from your local repository, without having to add the submitter’s repo as a
remote all the time. It is also nicely done in that it does not affect normal
clones in any way - unless you actually want to fetch them.

However, there is one case where it may have an undesired side effect: mirrors.
For example, I routinely mirror the Math.NET Numerics mainline repository to
a couple other places, including Codeplex, Gitorious and Google Code. I want
a mirror to exactly mirror the source repository, adding all new branches and
tags automatically, but also remove those that have been deleted in the source.
Git has excellent support for such exact mirroring. Unfortunately this mirroring
mechanism includes all the pull refs as well, which may not be what you want.
In Math.NET Numerics, some pull request actually base on an old (long removed)
branch that included some corrupt objects. So in this case, including them in the
mirror not only doubles the repository size, it also causes a corrupt git file system.

Luckily there is an easy way to skip them in the mirror, but to do that we must
understand how git refs actually work:

Likely the most requested feature for Math.NET Numerics is
support for some form of regression, or fitting data to a curve.
I’ll show in this article how you can easily compute regressions manually using Math.NET,
until we support it out of the box. We already have broad interpolation support,
but interpolation is about fitting some curve exactly through a given set of data points
and therefore an entirely different problem.

For a regression there are usually much more data points available than curve parameters,
so we want to find the parameters that produce the lowest errors on the provided data points,
according to some error metric.

In a recent post about new deployment and versioning approaches in Lokad.Cloud I mentioned that I’m also heavily refactoring the old cloud service framework and runtime. That refactoring was long due but also required to support these new approaches effectively.

In essence, developing Cloud Services still works as before. There is a framework library (Lokad.Cloud.Services.Framework) that provides base classes for a small set of service types that you can derive from. The following figure shows the dependencies of all involved components:

There is a lot of guidance out there on how to migrate from Mercurial to Git, but they often leave you with a repository in a bad state. Even more so if it originally was a subversion repository, then migrated to Mercurial and now finally to Git.

The Lokad.Cloud repository was such a case. The committers and authors in the commit history were a complete mess, but that’s not that much of an issue in practice. Worse is the fact that most text files were stored with CLRF line endings instead of LF internally. Git supports platform-native checkouts (CRLF on Windows, LF on Linux) quite nicely, but it only works well if text files are normalized to LF internally when committed. I strongly recommend doing that, as it will save you from a lot of trouble later on. Luckily it is also the default behavior for new repositories.