What’s coming for the Python 3.5 installer?

Last year at PyCon US, I volunteered to take over maintenance and development of the Python installers for Windows. My stated plan was to keep building the installer for Python 2.7 without modification, and to develop a new installer for Python 3.5. In this post, I’m going to show some of the changes I’ve been working on. Nothing is settled yet, and more changes are basically guaranteed before the first releases occur, but I’m happy that we’ll soon have a more powerful and flexible installer.

The installer will first be available for Python 3.5.0 alpha 1, due to be released in February.

Changes You Will Notice

The most dramatic change (and the most likely to be removed before the final release) is new default installation locations.

Installing a copy of Python for all users on a machine and allowing everyone to modify it (the default under Python 3.4 and earlier) is a massive security hole. When installed into Program Files, only administrators can modify the shared files, and so users are better protected from malicious or accidental modifications.

Those who have used the Just for Me option in previous versions of Python are likely to have been surprised when it did not work as expected. For Python 3.5, this is now a true per-user installation. All files are installed into a directory than can only be accessed by the current user and the installation will work without administrative privileges.

The first two buttons on this page are single-click installs, meaning you’ll get all the default features and options, including pip and IDLE. For most users, these will dramatically simplify the process of installing Python.

However, many of us (myself included) like to be a bit more selective when we install Python. The third button, Customize installation, is for us.

There are two pages of options. The first is a list of features that can be added or removed independently of the rest of the installation. Compared to the old-style tree view, the simple list of checkboxes makes it easier to see what each feature provides. This is also the screen you’ll see when you choose to modify an existing installation.

The second page is advanced options, including the install location which (currently) defaults to the legacy directory, allowing you to install Python 3.5 identically to the older versions with the same amount of clicking. Right now, the options are basically identical to previous versions, but they are no longer mixed up with installable features. The way they are implemented has also been improved to be more reliable.

From here, the rest of the installation proceeds as you’d expect. The final page retains the familiar message (thanks, Mark!) and also adds some links into the online documentation.

Changes You Will Not Notice

One interesting option you may have spotted on the Advanced Options page is a checkbox to install debugging symbols (.pdb files). These are handy if you work on or debug C extensions (for example, Python Tools for Visual Studio‘s mixed-mode C/Python debugging feature requires Python’s PDB files), and this is an easy way to install them. Previously the symbol files were available in a separate ZIP, but now they are just a checkbox away.

But wait, doesn’t this make the installer a larger download? Yes, or at least it would if the installer included the debugging symbols.

The biggest change to the installer is its architecture. Previously, it was a single MSI with a single embedded CAB that contained all the files. The new installer is a collection of MSIs (currently 12, though this could change), CABs (currently 16) and a single EXE. The EXE is the setup program shown above, while the CABs contain the install files and the MSIs have the install logic.

With this change, it is possible to lazily download MSIs and CABs as needed. Although it’s not marked in the screenshot above, the “Install debugging symbols” option will require an active internet connection and will download symbols on demand. In fact, it’s trivially easy to download all the components on demand, which reduces the initial download to less than 1MB.

My initial plan is to release four downloadable installers for Python 3.5.0 alpha1: two “web” installers (32-bit and 64-bit) and two “offline” installers that include the default components (download size is around 20MB, and it includes everything that was included in earlier versions). Depending on feedback and usage, this may change by the final release, but initially I want to offer useful alternatively without being too disruptive.

Another change that is part of the build process is code signing. Previously, only the installer was signed, which meant that undetectable changes could be made to python.exe or pythonXY.dll after installation. As part of reworking the installer, I’ve also moved to signing every binary that is part of a Python installation. This improves the level of trust for those who choose to validate signatures, as well as using the signed UAC dialog rather than the unsigned one when running Python as an administrator.

Changes For Administrators

For those who have scripted or automated Python installation from the old MSIs, things are going to change a bit. I believe these are for the better, as we never previously really documented and supported command-line installation, and I’ll be interested in the feedback from early adopters.

The first concern likely to arise is the web installers – how do I avoid downloading from the Python servers every time I install? What if I have to install on two hundred machines? Two thousand? The easiest way is to simply download everything once with the “/layout” option:

python-3.5.0a1.exe /layout \\shared\python\3.5.0a1

This will not install Python, but it will create a folder on a shared machine (or a local path) and download all the components into that folder. You can then run `python-3.5.0a1.exe` from that location and it will not need to download anything. Currently the entire layout is around 26MB for each of the 32-bit and 64-bit versions.

To silently install, you can run the executable with `/quiet` or `/passive`, and customisation options can be provided as properties on the command-line:

I’m not going to document the full list yet, as they may change up until the final release, but there will be a documentation page dedicated to installing and configuring Python on Windows.

How Can I Try This Out Early?

I’m still very actively working on this, but you can get my changes from hg.python.org/sandbox/steve.dower on the Installer branch. The build files are in Tools/msi and will (should) work with either Visual Studio 2013 or Visual Studio 2015 Preview.

Where Do I Complain About This?

I am keen to hear constructive feedback and concerns, so come and find the threads at python-dev. Nothing is unchangeable, and the Python community gets to have its say, though right now I’m looking to stabilise things up until alpha so please don’t be too upset if your suggestion doesn’t appear in the first release.

If you’re at all angry or upset, please make sure you’ve read the entire post before sharing that anger with everyone else. (That’s just general good advice.)

Hey there! I just wanted to say “Thanks” for taking this on and striving to make the experience for Windows based Pythonistas a better one. It’d be great if Python wasn’t the nightmare for Windows users Jessica McKellar talks about in her 2013 Keynote at the NZ Kiwi Pycon.

For me, I don’t really have any intentions of moving off 2.7 unless it’s to Nim. But why not just keep it really simple, offer installers for each version, with everything included. Essentially as archives for old versions. Then have 1 installer that in theory should never need updating ever. It always installs the last release, pulling an image online for the installer showing what it is, and including nothing. Pulls down the latest release and any optional selections.

Rather than 8 releases, 3 offline, 4 online and 1 with an Ask.com toolbar and Google Chrome Spyware.

The command will change, but it will still be possible via a Just for Me installation, where the only thing added outside the specified install directory is the program registration (for uninstall/upgrades). If you install, copy the directory tree, and then uninstall, there’ll be nothing left outside the directory you copied to. (FWIW, the command will be something like [“d:\python-3.5.0.exe”, “/quiet”, “InstallAllUsers=0”, “TargetDir=D:\distribution_tocreate”], subject to changes up until release – I’ll document all the options though.)

Afraid not, due to the way that Windows Installer works. It’s designed to install a product once across a system, which is what the main installer is meant to do. We want to know about the main installation so we can update (or not update) files with each new version.

Other installers are free to handle these update scenarios in their own way, including ignoring it completely, and the way to do this is to extract the official Python binaries from the installer (or rebuild/sign them), and the easiest way to do that is to install it Just for Me and copy the files from there.

I wouldn’t recommend automating the install-copy-uninstall sequence as part of your build anyway. It makes far more sense as a configuration step that all parallel builds on your machine can use and then you never need to uninstall. (Alternatively, if your build scripts don’t uninstall, subsequent scripts will very quickly get past the install step without modifying any files.)

I am excited about the web-based installers – I maintain Pynsist, a tool which builds application installers that include and automatically run the Python installer. This will allow it to make much smaller application installers without me implementing logic for a web-based installer myself.

If I run the installer in passive mode pointing at a directory where Python is already installed, will it be smart enough to avoid downloading components for files that are already installed?

Also very pleased to see that 32 and 64 bit Pythons will be installed into different locations; I was surprised to discover that wasn’t already the case.

The installer will only allow one installation per machine, so if the installer has been run on that machine already then it won’t attempt to install it again, regardless of the directory you request. The official installer is intended to be a system-wide Python, and it’s never really been capable of doing private installations (I’m not sure exactly what you’re asking for, but I know people want both).

If you want to embed the web installer and run it, I’d suggest leaving it at the default install directory and then reading the registry keys or using the py.exe launcher (which you could also make your own copy of if you need it in a different directory). However, if you really want to install a copy of Python just for the app that’s being installed, I’m afraid you’ll need to do most of the work yourself.

I am already working with systemwide Python installations, and using the py.exe and pyw.exe launchers to select the appropriate one for each application, so no problems there. With current versions of Python I change the default install directory so that 32 and 64 bit versions don’t collide, but with 3.5 I won’t need to do that.

It seems that to install, it requires elevated permissions. Is there a way to make it not ask for elevated permissions when I install to some specific directory which is not e.g. the C drive or program files? Because I’m trying to automate the installation from the command line and that stops the installation.

The Windows docs describe how you can automate installing from the command line, and InstallAllUsers=0 is an option that you’ll want.

That said, to install the C runtime package necessary for the latest versions requires administrative privileges. Eventually that package will be distributed via Windows Update and so everyone will have it automatically, but for now there’s no way around this limitation (IIRC, you can pass Include_crt=0 to skip it, but then Python won’t work, so there’s not much point doing this.)

Yeah, it’s not looking like mingw-w64 (nor any of the other compilers that try and offer MSVC compatibility) will be ready on day one. Nobody from Microsoft has really had time to try and fix other compilers (they’re working pretty hard on their own) so I wouldn’t expect any direct contributions.

That said, there have been plenty of preview releases and updates about how the new CRT will work, so somebody else could easily do it. And as I understand it, the main concern is licensing rather than code – if mingw-w64 supports the .lib file format then it should just work with the CRT libs that ship with Visual Studio. The problem is that it isn’t clear what the redistribution permissions are, and so most projects play it safe and refuse to redistribute those files and wait for people who have never had contact with them to reimplement them.

If you know one of the core maintainers of one of these projects, feel free to put them in touch with me and I’ll try and set up some conversations with our lawyers to see if we can figure out a way to make a better arrangement work.

I hope you will ship sqlite 3.8.10.1 for beta, as it includes a bunch of old bug fixes, I I read well D. Richard Hipp
““
At the request of a notable downstream, we are pushing the release of
SQLite 3.8.10 forward by a month or so. We will start a
release-testing cycle later today. A checklist is now online athttps://www.sqlite.org/checklists/3081000/index – when this checklist
goes all green, we will cut the release. (Previous checklists can be
seen at https://www.sqlite.org/checklists/index.html)

Source code snapshots of the first beta can be found at
(https://www.sqlite.org/download.html). We will probably be adding
some additional beta products (ex: DLLS) for testing over the next few
hours.

The 3.8.10 release consists mostly of performance enhancements and
fixes for arcane bugs found by the AFL fuzzer. The trunk has been
stable for weeks. We expect this to be a very trouble-free release.
If you have any concerns or questions about the current state of the
code, please bring them up as soon as possible.
““

This comment would have been better posted on bugs.python.org, but I have actually been working on upgrading to 3.8.11 this morning, so if all the tests are happy with it then the next round of releases should have that.

Thank you for the new installer! It is certainly is a welcome change. But there’s one thing I was hoping you could shed some light on.

I could extract the previous versions of the MSI installers (with MSIx from JSWare) which would give me the Python distribution with the directory structure preserved. This would allow for me to use it as a portable yet complete Python distribution (along with pip, a few packages installed through pip, and tkinter) that I could carry around on a USB drive as I switch between multiple PCs everyday.

This no longer seems to be the case though with the new installer. Is there a way to extract the Python distribution from the new installer or is there an /extract switch in the works that I could use to do what I could before?

The best way to do this (which will still result in a corrupt Python install, but if that’s what you want…) is to install Python once Just For Me, make a copy of the entire installation directory, and then uninstall the original.

It still won’t necessarily work properly on a machine that has another similar version of Python installed (though it will likely be better than 3.4 and earlier), and you will need to make sure that all your machines have the C Runtime installed manually, but otherwise it will be okay in many cases. If you add a pyvenv.cfg file containing applocal = True alongside the .exe file it might be safer, but this isn’t really the purpose of that flag.

When we discussed what the core Python teams responsibilities are here, it was decided that we don’t have to be the ones to provide a portable Python build. The simple and reliable per-user installation is within scope, but other setups are better handled by other groups.

There’s an issue on bugs.python.org working on this problem, but the current theory is that you need to install other Windows updates before you can install the CRT. Make sure you’re fully up to date (Vista and Win7 need service packs, Win8/8.1 need a cumulative update from April 2015).

Is there any chance of getting a normal MSI installer? This works fine if you’re manually running it, but is murder to distribute via SCCM (or any other software managment system, i imagine) . It returns odd error codes (-2016410844) that get interpreted as an install failure, and is a general nightmare to work with. I think it’s great that it’s per-user and you’re using MSI under the hood, but the generally accepted way to use MSI is one MSI file, with these other parts as features. Users see the same interface, and I can create transforms to select which options are installed for the user and I can reliably uninstall it on request.

I’m always considering it, and there’s a real possibility of releasing an admin-only MSI. My biggest concern is how that would interact with attempts to install from the EXE.

I’m thinking about writing a post (and potentially a conference talk) on the logic behind the different Python distribution formats to make this clearer, but you’ve accurately summed up the main (legitimate) use-case for a single MSI.

Hi Steve,
Any news on a monolithic MSI for use with SCCM? I’ve tried installing with /layout and an unattend.xml, but something odd happens and the install seems to roll back. Helpfully it works perfectly when manually run, so it must be something to do with the SYSTEM user SCCM uses to install software.

I did find a reference to using /layout and installing those MSIs individually passing TARGETPATH=location (Your reply to a bug ticket complaining about the new installer from 2015).
Does this still work and will it keep working in the future?
Are there other options I can or should pass to the MSI files, and finally, is there a preferred order to install the collection of MSI files in?
If this is a way forward for SCCM pending or instead of or until a single MSI is available, can you document it on the python for windows docs please?
Thanks,
Bryn

I’ve heard about the bug when running as SYSTEM but haven’t been able to get any details from anyone (some log files would be greatly appreciated!) or find the time to set up my own repro of it.

Did you see my latest post on this site where I discussed installers? There are a few serious incompatibilities that would result from using a monolithic MSI if any of your users also used the EXE on their machine, and though I suspect that would be unlikely I’m having a hard time just ignoring those problems. Otherwise, I simply don’t have time to develop and maintain another installer right now 🙁

I’m not recommending installing the MSIs individually anymore, but if you want to see what options to pass and what order, run an install with the exe and read the log file. However, a few features are installed by the executable rather than the MSIs, so you can’t necessarily fully reproduce an install directly anymore.