Windows Subsystem for Linux Overview

We recently announced Bash on Ubuntu on Windows which enables native Linux ELF64 binaries to run on Windows via the Windows Subsystem for Linux (WSL). This subsystem was created by the Microsoft Windows Kernel team and has generated a lot of excitement. One of the most frequent question we get asked is how is this approach different from a traditional virtual machine. In this first of a series of blog posts, we will provide an overview of WSL that will answer that and other common questions. In future posts we will dive deep into the component areas introduced.

Posted on behalf of Deepu Thomas.

History of Windows Subsystems

Since its inception, Microsoft Windows NT was designed to allow environment subsystems like Win32 to present a programmatic interface to applications without being tied to implementation details inside the kernel. This allowed the NT kernel to support POSIX, OS/2 and Win32 subsystems at its initial release.

Early subsystems were implemented as user mode modules that issued appropriate NT system calls based on the API they presented to applications for that subsystem. All applications were PE/COFF executables, a set of libraries and services to implement the subsystem API and NTDLL to perform the NT system call. When a user mode application got launched the loader invoked the right subsystem to satisfy the application dependencies based on the executable header.

Later versions of subsystems replaced the POSIX layer to provide the Subsystem for Unix-based Applications (SUA). This composed of user mode components to satisfy:

Process and signal management

Terminal management

System service requests and inter process communication

The primary role of SUA was to encourage applications to get ported to Windows without significant rewrites. This was achieved by implementing the POSIX user mode APIs using NT constructs. Given that these components were constructed in user mode, it was difficult to have semantic and performance parity for kernel mode system calls like fork(). Because this model relied on the need for programs to be recompiled it required ongoing feature porting and was a maintenance burden.

Over time these initial subsystems were retired. However, since the Windows NT Kernel was architected to allow new subsystem environments, we were able to use the initial investments made in this area and broaden them to develop the Windows Subsystem for Linux.

Windows Subsystem for Linux

WSL is a collection of components that enables native Linux ELF64 binaries to run on Windows. It contains both user mode and kernel mode components. It is primarily comprised of:

User mode session manager service that handles the Linux instance life cycle

It is the space between the user mode Linux binaries and the Windows kernel components where the magic happens. By placing unmodified Linux binaries in Pico processes we enable Linux system calls to be directed into the Windows kernel. The lxss.sys and lxcore.sys drivers translate the Linux system calls into NT APIs and emulate the Linux kernel.

Figure 1: WSL Components

LXSS Manager Service

The LXSS Manager Service is a broker to the Linux subsystem driver and is the way Bash.exe invokes Linux binaries. The service is also used for synchronization around install and uninstall, allowing only one process to do those operations at a time and blocking Linux binaries from being launched while the operation is pending.

All Linux processes launched by a particular user go into a Linux instance. That instance is a data structure that keeps track of all LX processes, threads, and runtime state. The first time an NT process requests launching a Linux binary an instance is created.

Once the last NT client closes, the Linux instance is terminated. This includes any processes that were launched inside of the instance including daemons (e.g. the git credential cache).

Pico Process

As part of Project Drawbridge, the Windows kernel introduced the concept of Pico processes and Pico drivers. Pico processes are OS processes without the trappings of OS services associated with subystems like a Win32 Process Environment Block (PEB). Furthermore, for a Pico process, system calls and user mode exceptions are dispatched to a paired driver.

Pico processes and drivers provide the foundation for the Windows Subsystem for Linux, which runs native unmodified Linux binaries by loading executable ELF binaries into a Pico process’s address space and executes them atop a Linux-compatible layer of syscalls.

System Calls

WSL executes unmodified Linux ELF64 binaries by virtualizing a Linux kernel interface on top of the Windows NT kernel. One of the kernel interfaces that it exposes are system calls (syscalls). A syscall is a service provided by the kernel that can be called from user mode. Both the Linux kernel and Windows NT kernel expose several hundred syscalls to user mode, but they have different semantics and are generally not directly compatible. For example, the Linux kernel includes things like fork, open, and kill while the Windows NT kernel has the comparable NtCreateProcess, NtOpenFile, and NtTerminateProcess.

The Windows Subsystem for Linux includes kernel mode drivers (lxss.sys and lxcore.sys) that are responsible for handling Linux system call requests in coordination with the Windows NT kernel. The drivers do not contain code from the Linux kernel but are instead a clean room implementation of Linux-compatible kernel interfaces. On native Linux, when a syscall is made from a user mode executable it is handled by the Linux kernel. On WSL, when a syscall is made from the same executable the Windows NT kernel forwards the request to lxcore.sys. Where possible, lxcore.sys translates the Linux syscall to the equivalent Windows NT call which in turn does the heavy lifting. Where there is no reasonable mapping the Windows kernel mode driver must service the request directly.

As an example, the Linux fork() syscall has no direct equivalent call documented for Windows. When a fork system call is made to the Windows Subsystem for Linux, lxcore.sys does some of the initial work to prepare for copying the process. It then calls internal Windows NT kernel APIs to create the process with the correct semantics, and completes copying additional data for the new process.

File system

File system support in WSL was designed to meet two goals.

Provide an environment that supports the full fidelity of Linux file systems

Allow interoperability with drives and files in Windows

The Windows Subsystem for Linux provides virtual file system support similar to the real Linux kernel. Two file systems are used to provide access to files on the users system: VolFs and DriveFs.

VolFs

VolFs is a file system that provides full support for Linux file system features, including:

Linux permissions that can be modified through operations such as chmod and chroot

Symbolic links to other files

File names with characters that are not normally legal in Windows file names

Interoperability between Windows applications and files in VolFs is not supported.

DriveFs

DriveFs is the file system used for interoperability with Windows. It requires all files names to be legal Windows file names, uses Windows security, and does not support all the features of Linux file systems. Files are case sensitive and users cannot create files whose names differ only by case.

All fixed Windows volumes are mounted under /mnt/c, /mnt/d, etc., using DriveFs. This is where users can access all Windows files. This allows users to edit files with their favorite Windows editors such as Visual Studio Code, and manipulate them with open source tools in Bash using WSL at the same time.

In future blog posts we will provide additional information on the inner workings of these component areas. The next post will cover more details on the Pico Process which is a foundational building block of WSL.

Deepu Thomas and Seth Juarez discuss the underlying architecture that enables the Windows Subsystem for Linux.

MOUNT_POINT=./mount
USB_DISK_MAJOR=b
USB_DISK_MINOR=1
if [ $(whoami) == “root” ] ; then
if [ ! -e $MOUNT_POINT ] ; then
mkdir $MOUNT_POINT
fi
sudo mount /dev/sd”$USB_DISK_MAJOR””$USB_DISK_MINOR” $MOUNT_POINT
# /dev/sda is your root device, if you only have one disk drive in your machine, with /dev/sda1 refering to partion 1 on your disk drive
#with this in mind, if you have only one disk drive in your machine, and you plug in a working, partitioned usb disk drive, with 2 paritions, you can run the command `ls /dev/sd*`. you should then see, a list displayed from your terminal the looks like “/dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb /dev/sdb1 /dev/sdb2”. your usb disk drive will be /dev/sdb, where /dev/sdb1 will be the first partition, and likewise, /dev/sdb2 will be the second partition on the disk.
else
echo “You must be root, or Sudo”
fi

## by the time that I wrote this, I came to my senses, and realized that the question asked was not directed for a baremetal linux os. Whoops! maybe some will find this useful anyways.

What does this mean? Will programs whose builds use “Makefile” and “makefile” to cater for different kinds of make break? Will archives containing such files unpack successfully? If users can’t make files like “File” and “file”, then why bother with case sensitive files?

DriveFs looks like just a thin wrapper around NTFS, so it has the same features/limitations (e.g. case preserving but non-sensitive).
If you try to build Linux apps under shared drives (/mnt/c etc) you’ll have issues.

If you use those in VolFs areas (/bin, /usr/, /etc, /home) it works fine as those have extra stuff behind the scenes to track and map the POSIX stuff over.

From the looks of it VolFs is in the AppData part of your user account, and while the files are there I’m guessing there’s a secondary store somewhere tracking the non-NTFS stuff.

I believe that’s meant to be “Files aren’t case sensitive”. You can for example use `cd /mnt/c/users/jessica/documents’ and it will open C:\Users\Jessica\Documents. Note the difference in case. There is a UserVoice request to have this behaviour changed, that people can vote on! Please vote if you can!!

> What does this mean? Will programs whose builds use “Makefile” and “makefile” to cater for different kinds of make break? Will archives containing such files unpack successfully? If users can’t make files like “File” and “file”, then why bother with case sensitive files?

I suspect that they meant case-preserving, case-insensitive, like the rest of Windows. If they do mean case-sensitive, then the restriction is most likely so that files made from this subsystem still plays nice with the rest of Windows.

Windows’ filesystem handling makes all filesystem access case insensitive (but case preserving, as NTFS itself has no issues with case sensitivity). This behaviour is old, and most likely only there for compatibility.

If they bypassed this in DriveFs, and let you create a Makefile and makefile side by side, then things like Explorer.exe would most likely break if you accessed that directory, as the regular API’s would consider those two files to be the same. You’re not sure which you would get a hold of if you tried to open one.

(I would argue that case insensitivity is a very bad idea, and only in place to allow software that is poorly written with inconsistent casing to run, but that’s a discussion for another day.)

This would cause an issue if you tried to do this on the mounted Windows drive (iirc at /mnt/c). However, for most purposes, you will want to work in a directory which is using VoIFs, not DriveFs, in which case everything will work as expected. The reason for this is that DriveFs manipulates a Windows filesystem, and since Windows does not permit files which only differ in case, allowing that on the Linux side would make all sorts of things potentially misbehave when accessing that directory from windows.

When DriveFS was initially designed we wanted to adhere to default Win32 behavior which is case preserving but case insensitive. Based on feedback from users who want to git clone/npm install to DriveFs locations we are looking into how we can bring more VolFs like behavior (symlinks, case sensitive files etc) to DriveFs in future builds.

The problem isn’t NTFS. NTFS has no problem with most linux filesystem behavior. The problem is with windows, which keeps a lot of what one might call “windows 95” behavior to make sure older programs don’t break. The same sort of issue applies to maximum path length. The old limit of ~250 characters is not imposed by NTFS, and there are some api calls available now (but apparently not used by the lxss developers) that don’t impose that limit. But lots of other windows programs (apparently including Explorer) don’t know how to handle them.

If you make a bunch of files in a DriveFS filesystem, you might expect to use them at some point with Windows applications; but if they rely on paths longer than the “old” limit, or case sensitivity, and windows couldn’t deal with them (e.g. backup programs, let alone Explorer or productivity or developer tools) you might not be happy. At the same time, the VolFS filesystem is advertised as not supporting interaction with Windows, so perhaps those limitations can be removed completely there?

I’d hope for two things: first, the ability to install the VolFS filesystem (or perhaps another VolFS filesystem, or even a volume formatted with a traditional Linux filesystem?) somewhere other than the %profile% on the system drive, which seems to never have enough storage space; and for the lxss developers to get the DriveFS filesystem as usable as possible by both environments.

Cygwin, SMB, virtual machine technologies (sharing “host” storage with “guest” machines), and windows drivers for ext2, have been coming up with compromises in these areas for quite a while now. I’d hope the WSL release would take the best from what those folks have learned over the years, instead of trying to reinvent the wheel.

That’s correct, you cannot have both filenames in the same directory, and note MacOS has the same restriction. Many wares are un-effected, others must be ‘ported’. You can run Qemu or Bochs or Virtualbox.

This so awesome, it means that never again do I have to deal with Virtualbox or VMWare for doing development and having strange file permissions issues.
It means I can keep my projects in C:/Workspace and mount them in /home/$USER/Workspace from where they can be picked up by apache, nginx, postgres, etc – their permissions being correct and not always showing up as executable crap in Whatnot VM platform ones uses is such an achievement.

I am still unable to understand if Docker can be used easily and how “fast” it is.
No matter what people say, for a developer, Docker is simply slow, snail slow on non Linux platforms such as OSX or Windows because it always needs a VM.
Maybe with WSL the Docker development experince will be as fast and snappy as on metal Linux.
I am not aware of OSX having such a feature, but they won developers harts due to its Unix similarities and good command line tools while still being a user friendly computer.
But Windows is catching up and it does it big time! Awesome times, wish it would had happened 10 years ago.

No. Virtualbox or Bochs can emulate hardware, ‘lxrun’ kits do not – they only emulate kernel calls and not all of them. The problem with Linux is poor hardware support and poor leverage* of the support it has. (* ex: driver has all features of hw enabled and software exists to expose it to end user and applications too *) The other problem is Linux OS distros tend to better support hardware that dis-includes USA products such as Windows 10, Apple iOS, features of Intel processors. Never assume there are no tech wars going on.

With reference to question on how you implemented lxss, you mentioned that it is based on Linux’s syscall interface. You cherry picked set of syscalls required by current set of supported PICO processes. While linux interface is unchanged for decades and kind of a choice over POSIX, it is neither a standard nor governed by a spec. There are many addons like ASLA. I guess applications depending on ASLA can not be run on WSL unless lxss extends the kind of clean room implementation mentioned in interview.

In that context couple questions

1. Yes Linux interface didn’t change for decades, but there is criticism around it by kernel developers themselves. How lxss plan to cope up with any changes Linux syscall interface brings up in future?
2. If I just do apt-get something that needs a specific syscall (say from an add on) which is not part of lxss and even NT kernel may not have a clue, what are the options? As a developer can I provide an addon to lxss? I mean can I configure lxsession manager to load a library from user code and hook with lxss as an extra catalogue when it looks for syscall translation to NT call.

Very interesting, thanks for the talk. The pico-process details will be very enlightening and help round-out the information, I’m sure.

How does starting bash start the session manager (which winds up starting the Linux instance, which winds-up loading /bin/bash)? Can you elucidate that chain of events?

Perhaps I should wait for the pico-process talk to be put online, but: What are the mechanics of requests flowing from the pico-process to LXCore/LXSS? Linux syscalls use syscall/sysenter, just like Windows. Does the pico-process TEB just point to a different system service table, and support for this table is what’s provided by the “pico-provider drivers”?

How does the fork()/clone() implementation work under the hood? Does it have the lightweight map-all-pages/copy-on-write semantic, or does it have the heavyweight copy-all-pages semantic that original UNIX had?

The limit is not there because of NTFS, it is there in the traditional Windows APIs to keep compatibility with older Windows. In fact, developers can now bypass that limit (though not without risking incompatibility with other Windows programs that still expect or impose the limit).

If metadata related access is stored as extended properties in NTFS and NT does not understand them in world of windows, how the access is maintained? example if I create a file using bash and assign read only attribute, will it be read only for windows? if I go to windows explorer? same way, if I create a file using windows, and mark it say read only, will it be read only for bash for linux on windows!!!?

Come on Satya, et al, you need to step up the game. Take Linux and make it yours, prod and push Microsoft into its next evolution, make your mark the way Jobs did when he ditched the OS 9 legacy and reinvented Apple. Most people would never notice if you started running Linux kernel instead of NT kernel and map NT atop Linux, i.e. “perfect” the WINE layer or something close to it, to provide a straightforward migration path away from NT kernel and onto Linux land… I mean, it can’t hurt Microsoft to re-take the Linux desktop and server market in one fell swoop just by providing it a nice GUI the way Jobs did while using 20+ years of an accumulated codebase, can it?

Can you please tell us if this is the equivalent in what wine is in linux?
Wine susposed do the “same” but in Linux for Windows Apps 🙂 So WSL is like that? Its 100% native way to run linux binaries on windows?

i think they will not do such of thing.. Wine struggles do the same without any help from Microsoft (how much good that was, if ms help wine to make more quick steps…). Is only one way i think :p Is not philanthropist foundation (ms), they will only take from Open Source, not give back 😛

Very nice post about linux subsystem. Today linux environment is using and orchestrating solutions with containers, like docker. There is a lot of environments using kubernetes, mesos and similar softwares to auto scale container environments. Now I see microsoft going into this game. I don’t see any restriction to Microsoft start many linux system and run dockers images directly. May be necessary implement some extra api (cgroup) but I think that in near future we will se Windows running docker images.
In fact , some time ago I heard microsoft interacting with docker team, and I think that the first result of this interaction is this linux subsystem.

I’ve turned my back on Windows after the 2000 version. I never looked back so far. My wife’s new convertible with Windows 10 is quick and shows touch integration that no Linux can match. And now I can bring all the tools that I love to use, and even install plain apt packages? I’m tempted to give it a try, and that’s huge change.

“Powershell” sounds like it came from a box of Cracker Jacks and is as logically programmable a secret decoder ring

“Bash” is macho and programmable. But, without the ability to interact with the core OS in an real, traditional UNIX type open standards based way (POSIX+), MS is just adding another box of Cracker Jacks to the Cracker Jacks.

I was an MS and UNIX junkie back to the days of DOS and the original Win SDK. If Ballmer hadn’t screwed up so bad, I’d probably still have Win as my primary dev environment. Unfortunately, that changed about 6 years ago.

IMO it seems a little strange to have each windows user have a separate instance of Ubuntu. wouldn’t it have made more sense to have a single shared ‘root’ instance and use PAM on the Ubuntu side to thunk authentication through Windows (bypassing the tradition /etc/passwd junk)?

Symbolic links is what’s stopping me from using this. In order to use windows app (editors, IDEs) we need to have our workspaces in /mnt/. But this is preventing me from being able to npm install anything on /mnt/ because of the symlink requirement.

Is there a way to mount a removable drive such as flash drives or network drives into the /mnt file structure.

Can multiple Windows IDs share the same root file system (for instance, I want to have the same set of software packages on my local ID and Microsoft account.

I already put this other bug in via feedback a couple of weeks ago, but the bug’s still there, so not sure if anyone looked at it yet, but you get a Blue Screen crash and restart if you have ProxyCap enabled (for Socks access through a VPN), and you try to access one of the Socks-enabled servers from any Bash networking tool, for instance ssh or ping.
I haven’t tried the socks competitor OpenText with bash yet, but plan to do that soon.

We are developing Python IDE (PyCharm) and we need some way to launch python in WSL. I have several ideas, but all of them look hackerish. What is the best way to do that?

* Ideally I would like to do “bash.exe -c ..”, but I can’t since piping does not work.
* I can use ssh, I even was able to launch sshd (with password-based (interactive) auth) but sshd support seems to be buggy (try to launch it and connect with, say, putty).
* It would be ineteresting to launch Pico process directly, and there are some functions (starting with “Psp”), but they are not public, are they?
* I can launch bash on console and connect to this console and use ConsoleAPI, but not sure if it may work.
* I can run python and send its output to network (using nc for example) but it does not look good, anyway.
* “CreateProcess” does not work with ELFs (only PE/COFF are supported, I believe)

So, what is the best way to achieve it? How can I launch WSL python from my Windows app and read its output interactively?

I have one question 🙂 i have not find at good quality informations about it searching on the internet, so i give me shot here 🙂

Windows Subsystem for Linux (WSL) can be used with the upcoming Windows Server 2016? It is running right now with the technical previews? There is your plan include that functionality to the Windows Server? (it makes sense that in Windows Server. A lot of developers and companies use Windows Server for development, servicing, etc. WSL is for development purposes so thats why i find it could fit in Windows Server)

Im trying to find any infos about those things on internet, but i have not any luck yet 🙂

How do I mount a network drive to the Linux subsystem? Using windows I can see V:\ without problems. Once I enter bash I cannot see it, how can I mount /mnt/v so that it points to the drive that I see in windows as V:\ ?

tried enabling it on windows 10 anniversary update windows 10 Version 1607 (OS Build 10.0.14393). But can’t.
Trying to switch on developer mode it wants windows updates run and installed. But the windows updates don’t run.
I am in the UK and my region, date and time settings are all correct.
But windows updates reports: “We couldn’t install updates because there’s a problem with the date and time information on your device. Make sure your date, time, and time zone settings are correct and we’ll try again later.”

Well, this feature is nice, but what is a point? I don’t believe that all nice Linux kernel features will be implemented and some features, that are emulated (like fork call) could cause a performance bottleneck, because it’s expected to be fast. I doubt that docker would be able to run on Windows natively soon and I doubt that anyone even need this. It’s definetely wouldn’t be used by anyone for production, since we already have Linux, that is already good in running Linux applications. It’s useless for casual Windows user since most of the Linux software is Open Source and most of it is already crossplatform. I don’t understand why Microsoft even put any effort in it? It would probably die in a few years like Windows Subsystem For Unix or however that thing was called.

At my work we build programs for MacOSX, Linux and Windows. If we could reduce the number of operating system we support that would be a free lunch for us in terms of productivity. The Linux version of our programs appear to run at native speed compared to a Linux PC. The implementation of the fork() call also seems to work well in our programs running on WSL. If we don’t need to maintain to ways of doing multiprocessing this would be a clear benefit for us.

Is there some reason that an ELF loader for Windows was not implemented, and that a special “pico-process” loader must be created for every Linux executable? Presumably this exists, like many of the other arbitrary limitations, to prevent integration and interoperability with the Windows host?

One has to question the usefulness of a subsystem that blocks interoperability with the host at every turn.
No arbitrary binaries
No host integration
No sockets
No services

What can one actually do with WSL? Is there a single substantial Unix application that can run in this environment or is it intended only for running shells and the GNU core utilities?

It is very disappointing that Microsoft did not include the Windows Subsystem for Linux in Windows Server 2016. I am back to install Cygwin instead of the fantastic Ubuntu for Windows. How many times Microsoft disappoints people like me? Every other day. They do not understand that we live in both worlds, and wish to unify them.

Thanks for your comment. I was not involved in this decision but I can speak to the rationale behind it. Since WSL is still in beta it probably isn’t the best fit for an enterprise / datacenter focused SKU. As WSL becomes more complete I suspect this decision will be revisited.

It is up to us as the administrators to decide if a product is appropriate for production or not. Surely Microsoft knew there would be demand for WSL on 2016. To cover their posteriors, all Microsoft need to have done was have a BIG DISCLAIMER that WSL isn’t ready for production and will not be supported.

One of the great advantages of the majority of open source projects is that they treat us as responsible entities and offer up their projects before they are “production ready. ” This is one of the best ways to get feedback from their potential user community.

Does this work for Redhat/CentOS? IF so 6.x or 7.x
2007-2008 I remember WSL by Microsoft Research. Is this the same?
I remember Microsoft winning the Linux World award running apps faster on WSL(2007-2008).

Great stuff. I’ve been working with it this week and can see this as an alternative to dual-boot machines we have now throughout the university. I have run into one issue, however. After sysprep and during OSD (using SCCM), I get 80070005 failure during applying image step. It has issues with permissions on some files in the lxss folder. Is there any way to include the already-configured WSL setup on an image? What is the preferred/planned deployment method?

I wanted to put lxss onto my school computer that I bought because it uses emmc and has limited space and all I need is bash, but microshaft winblows strikes again! You have to pay $100 to upgrade from windows 10 home to windows 10 pro, just to use bash as a developer; that is what upsets me Linux is free, there is no reason you should have to pay for it whatsoever if you already have a copy of windows 10, if they are going to implement it they should, period!

It is a good system, and I want it to get better. I am following the
“https://www.suse.com/communities/blog/make-windows-green-part-1/”
and the comman line is really indistinguishable from opensuse Leap 42.2 that I dual boot with (sometimes it’s duel) and
sudo zypper install xterm – gives an xterm window.

Like Hannes Kühnemund says in his blog
“Now close the SUSE bash, open it again and type “xterm”. The xterm application window should now appear on your Windows 10 desktop. Isn’t that cool?”

Would it be possible in the future for windows programs, such as IDEs, to run Linux programs installed in WLS.
For example, lets say my IDE requires node and I have it installed on WSL, it would be nice for the IDE to be able to use the node in WLS so I don’t have to install node for windows.

Since, WSL and windows are now interoperable having windows application silently use compatible Linux programs like node or git would be really great and eliminate the need to download separate versions of the programs for windows.

One of the coolest pieces of software from MS! I was stunned that our ConceptBase.cc ELF binaries run unmodified under WSL. I had never expected this since our software components is compiled from half a dozen different programming languages (C, C++, Java, Prolog, …). Amazing job and I like the path via bash as well.