Monday, December 28, 2015

Welcome to the one hundred and fourth issue of LLVM Weekly, a weekly
newsletter (published every Monday) covering developments in LLVM, Clang, and
related projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, December 21, 2015

Welcome to the one hundred and third issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, December 14, 2015

Welcome to the one hundred and second issue of LLVM Weekly, a weekly
newsletter (published every Monday) covering developments in LLVM, Clang, and
related projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, December 7, 2015

Welcome to the one hundred and first issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, November 30, 2015

We have been working hard for a few months now to rewrite the ELF support in lld, the LLVM linker. We are happy to announce that it has reached a significant milestone: it is now able to bootstrap LLVM, Clang, and itself and pass all tests on x86-64 Linux and FreeBSD with the speed expected of an LLVM project.

ELF is the standard file format for executables on Unix-like systems, such as Linux and BSDs. GNU ld and GNU gold are commonly used linkers for such systems today. In many use cases, the linker is a black box for which only speed matters. Depending on program size, linking a program takes from tens of milliseconds to more than a minute. We designed the new linker so that it runs as fast as possible. Although no serious benchmarking or optimization has been conducted yet, it is consistently observed that the new lld links the LLVM/Clang/lld executables in about half the time of GNU gold. Generated executables are roughly the same size. lld is not at feature parity with gold yet, so it is too early to make a conclusion, but we are working hard to maintain or improve lld’s speed while adding more features.

lld is command-line compatible with GNU ld so that it can be used as a drop-in replacement. This does not necessarily mean that we are implementing all the features of the GNU linkers in the same way as they did. Some features are no longer relevant for modern Unix-like systems and can be removed. Some other features can be implemented in more efficient ways than those in the traditional linkers. Writing a new linker from scratch is a rare occasion. We take advantage of this opportunity to simplify the linker while keeping compatibility with the existing linkers for normal use.

The new ELF linker is a relatively small program which currently consists of about 7000 lines of C++ code. It is based on the same design as the PE/COFF (Windows) support in lld, so the design document for the PE/COFF support is directly applicable to the ELF support.

The older ELF support still exists in lld repository in parallel with the new one. Please be careful to not confuse the two. They are separated at the top directory and do not share code. You can run the new linker with ld.lld command or by passing -fuse-ld=lld to Clang when linking.

We are still working on implementing remaining functionality such as improved linker script support or improved support for architectures beyond x86_64. If you are interested in the new linker, try it out for yourself.

Welcome to the one hundredth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Eagle-eyed readers will note we've now reached issue 100, marking 100 weeks of
uninterrupted service and of course meaning there's just 28 weeks to go until
an important numerical milestone.

Monday, November 23, 2015

Welcome to the ninety-ninth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, November 16, 2015

Welcome to the ninety-eighth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

This week's issue comes to you from Vienna where I'm just about to head home
from a short break (so apologies if it's a little later than usual and perhaps
a little less detailed). I'll admit that nobody has actually written in to beg
that LLVM Weekly share travel tips, but I will say that Vienna is a beautiful
city that's provided lots to do over the past few days. If you're visiting, I
can strongly recommend Salm Bräu for good beer and food.

Thursday, November 12, 2015

LLVM provides many useful command line tools to handle bitcode: opt is the most widely known and is used to run individual passes on an IR module, and llc invokes the backend to generate an assembly or object file from an IR module. Less known but very powerful is bugpoint, the automatic test case reduction tool, that should be part of every developer's toolbox.

The bugpoint tool helps to reduce an input IR file while preserving some interesting behavior, usually a compiler crash or a miscompile. Multiple strategies are involved in the reduction of the test case (shuffling instructions, modifying the control flow, etc.), but because it is oblivious to the LLVM passes and the individual backend specificities, "it may appear to do stupid things or miss obvious simplifications", as stated in the official description. The documentation gives some insights on the strategies that can be involved by bugpoint, but the details are beyond the scope of this post.

Read on to learn how you can use the power of bugpoint to solve some non-obvious problems.

Monday, November 9, 2015

Welcome to the ninety-seventh issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, November 2, 2015

Welcome to the ninety-sixth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, October 26, 2015

Welcome to the ninety-fifth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone
else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, October 19, 2015

Welcome to the ninety-fourth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone
else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or
@llvmweekly or @asbradbury on Twitter.

A good time was had by all at ORConf last week
at CERN. We had over 100 open source hardware enthusiasts join us in Geneva.
You can find my slides updating on lowRISC
here.
Videos should appear on youtube in the next week or so.

Monday, October 12, 2015

Welcome to the ninety-third issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone
else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Apologies that this week's issue comes rather late in the day, my laptop gave
up the ghost over the weekend while I was travelling leaving me with no way to
write it. Now I'm back, I've managed to dust off an old desktop from my closet
to write this issue (and to keep my unbroken streak). LLVM Weekly has been
sent out every Monday since it started on the first Monday of January 2014.
This weekend I was talking about lowRISC at ORConf
2015. You can find my slides
here.
There was a wide array of talks on open source hardware, including many on
lowRISC and RISC-V. The videos should hopefully be posted in the next week or
so.

Monday, October 5, 2015

Welcome to the ninety-second issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone
else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, September 28, 2015

Welcome to the ninety-first issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone
else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, September 21, 2015

Welcome to the ninetieth issue of LLVM Weekly, a weekly newsletter (published
every Monday) covering developments in LLVM, Clang, and related projects.
LLVM Weekly is brought to you by Alex Bradbury.
Subscribe to future issues at http://llvmweekly.org and pass it on to anyone
else you think may be interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, September 14, 2015

Welcome to the eighty-ninth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, September 7, 2015

Welcome to the eighty-eighth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Tuesday, September 1, 2015

This release contains the work of the LLVM community over the past six months: full OpenMP 3.1 support (behind a flag), the On Request Compilation (ORC) JIT API, a new backend for Berkeley Packet Filter (BPF), Control Flow Integrity checking, as well as improved optimizations, new Clang warnings, many bug fixes, and more.

Monday, August 31, 2015

Welcome to the eighty-seventh issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

It's a bank holiday weekend here in the UK, so apologies that you're reading
this a few hours later than usual. As a quick reminder, if you're able to be
in Geneva for the 9th-11th October then you should definitely come along to
ORConf.

Monday, August 24, 2015

Welcome to the eighty-sixth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Thursday, August 20, 2015

The LLVM Foundation is proud to announce it has been officially approved as a public charity with tax-exempt status under Section 501(c)(3) of the United States Internal Revenue Code. Contributions donated to the LLVM Foundation are fully tax deductible, retroactive to the organization establishment date of May 5, 2014.

The LLVM Foundation's primary mission is to provide accessible and informative educational tools for the LLVM Project and compiler technology to the general public. These educational tools include events such as the annual LLVM Developers' meeting. The LLVM Foundation also gives grants or scholarships to other nonprofit organizations and individuals (such as student travel to LLVM Foundation events). Lastly, the LLVM Foundation funds the infrastructure necessary to support the LLVM Project.

We hope to begin accepting donations online before the end of the year. If you are interested in contributing, please contact your employer: they may be willing to match contributions to a 501(c)(3) charity.

The IRS letter granting the LLVM Foundation 501(c)(3) tax-exempt status is available upon request until we can get it posted online. Please contact Tanya Lattner (tanyalattner@llvm.org), President of the LLVM Foundation.

Monday, August 17, 2015

Welcome to the eighty-fifth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, August 10, 2015

Welcome to the eighty-fourth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, August 3, 2015

Welcome to the eighty-third issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, July 27, 2015

Welcome to the eighty-second issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

I'd just like to highlight how much I really do appreciate people sending me
links for inclusion, e.g. LLVM-related blog posts or new releases of software
using LLVM (feature releases rather than simple bugfix updates). I'm not
omniescent - if an interesting blog post or software release goes unmentioned
here, I probably just didn't know about it!

Monday, July 20, 2015

Welcome to the eighty-first issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

I'm "on holiday" (at EuroPython) this week in Bilbao, mostly helping out the
Raspberry Pi team with the education track. Do say hello, particularly if you
want to chat lowRISC, LLVM, or Raspberry Pi.

Monday, July 13, 2015

Welcome to the eightieth issue of LLVM Weekly, a weekly newsletter (published
every Monday) covering developments in LLVM, Clang, and related projects.
LLVM Weekly is brought to you by Alex Bradbury.
Subscribe to future issues at http://llvmweekly.org and pass it on to anyone
else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or

Monday, July 6, 2015

Welcome to the seventy-ninth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Last week I was in Berkeley for the second RISC-V conference. If you weren't
able to make it, worry not because I liveblogged both day
one and
day
two.

Monday, June 29, 2015

Welcome to the seventy-eighth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

I'm in the Bay Area this week for the second RISC-V
workshop where my colleague and I
will of course be talking about lowRISC. If you're
not able to make it, keep an eye on the lowRISC blog which I intend to keep
updating semi-live with notes from the talks and presentations.

Monday, June 22, 2015

Welcome to the seventy-seventh issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

I'll be in California next week for the second RISC-V
workshop. Me and my colleague Wei
will both be giving talks about recent lowRISC
progress. Say hi if you're going to be there. I might have some spare time
towards the end of the week too if anyone wants to meet up.

Monday, June 15, 2015

Welcome to the seventy-sixth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, June 8, 2015

Welcome to the seventy-fifth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, June 1, 2015

Welcome to the seventy-fourth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

You may be interested in the second RISC-V
workshop,
which will be held in Berkeley June 29-30. Early bird registration ends today,
but academics can register for free. My colleague Wei and I will be there
representing lowRISC.

Monday, May 25, 2015

Welcome to the seventy-third issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Friday, May 22, 2015

OpenMP
support in Clang compiler is completed! Every pragma and clause from 3.1 version of the standard is supported in full, including combined directives (like ‘#pragma omp parallel
for’ and ‘#pragma omp parallel sections’). In addition,
some elements of OpenMP 4.0 are supported as well. This includes “almost
complete” support for
‘#pragma omp simd” and full support for ‘#pragma omp atomic’ (combined pragmas and a couple of clauses are still missing).

OpenMP
enables Clang users to harness full power of modern multi-core processors with
vector units. Pragmas from OpenMP 3.1 provide an industry standard way to
employ task parallelism, while ‘#pragma omp simd’ is a simple yet flexible
way to enable data parallelism (aka vectorization).

You will
see more than one “Hello” line with different thread numbers (note that the
lines may be mixed together). If you see only one line, try setting the
environment variable OMP_NUM_THREADS to some number (say 4) and try again.

Hopefully,
you will enjoy using OpenMP and witness dramatic boosts of your applications’
performance!

Monday, May 18, 2015

Welcome to the seventy-second issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, May 11, 2015

Welcome to the seventy-first issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, May 4, 2015

Welcome to the seventieth issue of LLVM Weekly, a weekly newsletter (published
every Monday) covering developments in LLVM, Clang, and related projects.
LLVM Weekly is brought to you by Alex Bradbury.
Subscribe to future issues at http://llvmweekly.org and pass it on to anyone
else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, April 27, 2015

Welcome to the sixty-ninth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Tuesday, April 21, 2015

Welcome to the sixty-eighth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Tuesday, April 14, 2015

The LLILC project (we pronounce it "lilac") is a new effort started at Microsoft to produce MSIL code generators based on LLVM and targeting the open source dotnet CoreCLR. We are envisioning using the LLVM infrastructure for a number of scenarios, but our first tool is a Just in Time (JIT) compiler for CoreCLR. This new project is being developed on GitHub and you can check it out here. The rest of this post outlines the rational and goals for the project as well as our experience using LLVM.

Why a new JIT for CoreCLR?

While the CoreCLR already has JIT, we saw an opportunity to provide a new code generator that has the potential to run across all the targets and platforms supported by LLVM. To enable this, as part of our project we're opening an MSIL reader that operates directly against the same common JIT interface as the production JIT (RyuJIT). This new JIT will allow any C# program written for the .NET Core class libraries to run on any platform that CoreCLR can be ported to and that LLVM will target.

There are several ongoing efforts to compile MSIL in the LLVM community, SharpLang springs to mind. Why build another one?

When we started thinking about the fastest way to get a LLVM based code generation working we looked around at the current open source projects as well as the code we had internally. While a number of the OSS projects already targeted LLVM BitCode, no one had anything that was a close match for the CoreCLR interface. Looking at our options it was simplest for us to refactor a working MSIL reader to target BitCode then teach a existing project to support the contracts and APIs the CoreCLR uses for JITing MSIL. Using a existing MSIL reader let us quickly start using a number of building-block components that we think the community can take advantage of. This fast bootstrap for C# across multiple platforms was the idea that was the genesis of this project and the compelling reason to start a new effort. We hope LLILC will provide a useful example - and reusable components - for the community and make it easier for other projects to interoperate with the CoreCLR runtime.

Why LLVM?

Basically we think LLVM is awesome. It's already got great support across many platforms and chipsets and the community is amazingly active. When we started getting involved, just trying to stay current with the developer mailing list was a revelation! The ability for LLVM to operate as both a JIT and as an AOT compiler was especially attractive. By bringing MSIL semantics to LLVM we plan to construct a number of tools that can work against CoreCLR or some sub set of its components. We also hope the community will produce tools what we haven't thought of yet.

Tool roadmap

CoreCLR JIT

Just In Time - A classic JIT. This is expected to be throughput-challenged but will be correct and usable for bringup. Also possible to use with more optimization enabled as a higher tier JIT

Install-time JIT - What .NET calls NGen. This will be suitable for install-time JITing (LLVM is still slow in a runtime configuration)

Ahead of Time compiler. A build lab compiler that produces standalone executables, using some shared components from CoreCLR. The AOT compiler will be used to improve startup time for important command line applications like the Roslyn Compiler.

The LLIC JIT will be a functionally correct and complete JIT for the CoreCLR runtime. It may not have sufficient throughput to be a first-tier jit, but is expected to produce high-quality code and so might make a very interesting second-tier or later JIT, or a good vehicle for prototyping codegen changes to feed back into RyuJIT.

What's Actually Working

Today on Windows we have the MSIL reader & LLVM JIT implemented well enough to compile a significant number of methods in the JIT bring up tests included in CoreCLR. In these tests we compile about 90% the methods and then fall back to RyuJIT for cases we can't handle yet. The testing experience is pretty decent for developers. The tests we run can be seen in the CoreCLR test repo.

We've establish builds on Linux and Mac OSX and are pulling together mscorlib, the base .NET Core library from CoreFx, and test asset dependencies to get testing off-the-ground for those platforms.

All tests run against the CoreCLR GC in conservative mode - which scans the frame for roots - rather than precise mode. We don't yet support Exception Handling.

Architecture

Philosophically LLILC is intended to provide a lean interface between CoreCLR and LLVM. Where possible we rely on preexisting technology.

For the JIT, when we are compiling on demand, we map the runtime types and MSIL into LLVM BitCode. From there compilation uses LLVM MCJIT infrastructure to produce compiled code that is output to buffers provided by CoreCLR.

Our AOT diagram is more tentative but the basic outline is that the code generator is driven using the same interface and the JIT but that there is a static type system behind it and we build up a whole program module with in LLVM and run in a LTO like mode. Required runtime components are emitted with the output objs and the platform linker then produces the target executable. There are still a number of open questions around issues like generics that need resolution but this is our first stake in the ground.

Experience with LLVM

In the few months we've been using LLVM, we've had a really good experience but with a few caveats. Getting started with translating to BitCode has been a very straightforward experience and ramp-up time for someone with compiler experience has been very quick. The MCJIT, which we started with for our JIT, was easy to configure and get code compiled and returned to the runtime. Outside of the COFF issue discussed below, we only had to make adjustments in configuration or straightforward overrides of classes, like EEMemoryManager, to enable working code. Of the caveats, the first was simple, but the other two are going to require sustained work to bring up to the level we'd like. The first issue was a problem with Windows support in the DynamicRuntime of the MCJIT infrastructure. The last two, Precise Garbage Collection, and Exception Handling, arise because of the different semantics of managed languages. Luckily for us, people in the community have already started working in these areas so we don't have to start from zero.

COFF dynamic loader support

One of the additions we made to LLVM to unblock ourselves was to implement a COFF dynamic loader. (The patch to add RuntimeDyldCoff.{h,cpp} is through review and has been committed). This was the only addition we directly had to make to LLVM to enable bring-up of the code generator. Once this is in, we see a number of bugs in the database around Windows JIT support that should be easier to close.

Precise Garbage Collection

Precise GC is central to the CoreCLR approach to memory management. Its intent is to keep the overhead of managed memory as low as possible. It is assumed by the runtime that the JIT will generate precise information about the GC ref lifetimes and provide it with the compiled code for execution. To support this we're beginning to use the StatePoint approach, with additions to convert the standard output format to the custom format expected by CoreCLR. We share some of the same concerns that Philip Reames wrote about in the initial design of StatePoints. E.g. preservation of "GCness" through the optimizer is critical, but must not block optimizer transformations. Given this concern one of our open questions is how to enable testing to find GC holes that creep in, but also enable extra checking that can be opted into if the incoming IR contains GC pointers.

There is a more detailed document included in our repo that outlines our more-specific GC plans here.

Exception Handling

The MSIL EH model is specific to the CLR as you'd expect, but it descends in part conceptually from Windows Structured Exception Handling (SEH). In particular, the implicit exception flow from memory accesses to implement null checks, and the use of filters and funclets in the handling of exceptions, mirrors SEH (here. is an outline of C# EH) Our plans at this point are to add all checks required by MSIL as explicit compare/branch/throw sequences to better match C++ EH as well as building on the SEH support currently being put into Clang. Then, once we have correctness, see if there is a reasonable way forward that improves performance.

Like GC, there's a detailed doc outlining our specific issues and plans in the repo here

Future Work

More platforms. Today we're running on Windows and starting to build for Linux and Mac OSX. We'd like more.

Complete JIT implementation

More MSIL opcodes supported

Precise GC support

EH support

Specialized memory allocators for hosted solutions. CoreCLR has been used as a hosted solution (run in process by other programs) but to support this we need a better memory allocation story. The runtime should be able to provide a memory allocator that is used for all compilation.

AOT - Fully flesh out the AOT story.

Links

Monday, April 13, 2015

Welcome to the sixty-seventh issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

EuroLLVM is going on today and tomorrow in
London. I hope to see a number of you there. Provided there's a reasonable
internet connection, I hope to be live-blogging the event on the
llvmweekly.org version of this issue.

Thursday, April 9, 2015

Fuzzing (or fuzz testing) is becoming increasingly popular. Fuzzing Clang and fuzzing with Clang is not new: Clang-based AddressSanitizer has been used for fuzz-testing the Chrome browser for several years and Clang itself has been extensively fuzzed using csmithand, more recently, using AFL. Now we’ve closed the loop and started to fuzz parts of LLVM (including Clang) using LLVM itself.

LibFuzzer, recently added to the LLVM tree, is a library for in-process fuzzing that uses Sanitizer Coverage instrumentation to guide test generation. With LibFuzzer one can implement a guided fuzzer for some library by writing one simple function:

extern "C" void TestOneInput(const uint8_t *Data, size_t Size);

We have implemented two fuzzers on top of LibFuzzer: clang-format-fuzzer and clang-fuzzer. Clang-format is mostly a lexical analyzer, so giving it random bytes to format worked perfectly and discovered over 20 bugs. Clang however is more than just a lexer and giving it random bytes barely scratches the surface, so in addition to testing with random bytes we also fuzzed Clang in token-aware mode. Both modes found bugs; some of them were previously detected by AFL, some others were not: we’ve run this fuzzer with AddressSanitizer and some of the bugs are not easily discoverable without it.

Just to give you the feeling, here are some of the input samples discovered by the token-aware clang-fuzzer starting from an empty test corpus:

static void g(){}

signed*Qwchar_t;

overridedouble++!=~;inline-=}y=^bitand{;*=or;goto*&&k}==n

int XS/=~char16_t&s<=const_cast<Xchar*>(thread_local3+=char32_t

Fuzzing is not a one-off thing -- it shines when used continuously. We have set up a public build bot that runs clang-fuzzer and clang-format-fuzzer 24/7. This way, the fuzzers keep improving the test corpus and will periodically find old bugs or fresh regressions (the bot has caught at least one such regression already).

The benefit of in-process fuzzing compared to out-of-process is that you can test more inputs per second. This is also a weakness: you can not effectively limit the execution time for every input. If some of the inputs trigger superlinear behavior, it may slow down or paralyze the fuzzing. Our fuzzing bot was nearly dead after it discovered exponential parsing time in clang-format. You can workaround the problem by setting a timeout for the fuzzer, but it’s always better to fix superlinear behavior.

It would be interesting to fuzz other parts of LLVM, but a requirement for in-process fuzzing is that the library does not crash on invalid inputs. This holds for clang and clang-format, but not for, e.g., the LLVM bitcode reader.

Help is more than welcome! You can start by fixing one of the existing bugs in clang or clang-format (see PR23057, PR23052 and the results from AFL) or write your own fuzzer for some other part of LLVM or profile one of the existing fuzzers and try to make it faster by fixing performance bugs.

Fuzz testing, especially coverage-directed and sanitizer-aided fuzz testing, should directly compliment unit testing, integration testing, and system functional testing. We encourage everyone to start actively fuzz testing their interfaces, especially those with even a small chance of being subject to attacker-controlled inputs. We hope the LLVM fuzzing library helps you start leveraging our tools to better test your code, and let us know about any truly exciting bugs they help you find!

Monday, April 6, 2015

Welcome to the sixty-sixth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, March 30, 2015

Welcome to the sixty-fifth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, March 23, 2015

Welcome to the sixty-fourth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, March 16, 2015

Welcome to the sixty-third issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, March 9, 2015

Welcome to the sixty-second issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, March 2, 2015

Welcome to the sixty-first issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Friday, February 27, 2015

This release contains the work of the LLVM community over the past six months: many many bug fixes, optimization improvements, support for more proposed C++1z features in Clang, better native Windows
compatibility, embedding LLVM IR in native object files, Go bindings, and more. For details, see the release notes [LLVM, Clang].

Many thanks to everyone who helped with testing, fixing, and getting the release into a good state!

Special thanks to the volunteer release builders and testers, without whom this release would not be possible: Dimitry Andric, Sebastian Dreßler, Renato Golin, Sylvestre Ledru, Ben Pope, Daniel Sanders, and
Nikola Smiljanić!

If you have any questions or comments about this release, please contact the community on the mailing lists. Onwards to 3.7!

Monday, February 23, 2015

Welcome to the sixtieth issue of LLVM Weekly, a weekly newsletter (published
every Monday) covering developments in LLVM, Clang, and related projects.
LLVM Weekly is brought to you by Alex Bradbury.
Subscribe to future issues at http://llvmweekly.org and pass it on to anyone
else you think may be interested. Please send any tips or feedback to
asb@asbradbury.org, or @llvmweekly or @asbradbury on Twitter.

Monday, February 16, 2015

Welcome to the fifty-ninth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or
@llvmweekly or @asbradbury on Twitter.

Monday, February 9, 2015

Welcome to the fifty-eighth issue of LLVM Weekly, a weekly newsletter
(published every Monday) covering developments in LLVM, Clang, and related
projects. LLVM Weekly is brought to you by Alex
Bradbury. Subscribe to future issues at
http://llvmweekly.org and pass it on to anyone else you think may be
interested. Please send any tips or feedback to asb@asbradbury.org, or
@llvmweekly or @asbradbury on Twitter.