Archive for May, 2018

Three weeks from our last update on this blog. Time flies when you are busy! As you know, openSUSE Leap 15.0 was released in the meantime, which also means the active development of SLE15 is coming to an end… so time to look a little bit further into the future.

That’s why we had a face-to-face workshop with the whole YaST Team at the beautiful city of Prague during several days right before joining the openSUSE Conference 2018.

But we have done much more in three weeks than attending workshops and conferences. Apart from last-minute fixes, here you have a list of some interesting changes we have done in YaST in this period. Take into account that some of these changes didn’t make it into Leap 15.0, although all will be available in SLES15 and are probably already integrated into openSUSE Tumbleweed.

Fine tuning installer behavior in small disks

As you may know, the default installation of SLE and both openSUSE distribution enables Btrfs snapshots in the root partition alongside separate partitions for /home and swap. That means a default installation needs quite some space. In SLE12 and openSUSE Leap 42.X, if such disk space was not there the installer silently tries to disable the separate /home and even the snapshots in order to be able to create an initial proposal.

That behavior has become configurable for each product and role with Storage-ng and during the last sprint there was some controversy about what the configuration should be, both for openSUSE and the SLE family. It may look like a minor problem, but it becomes very relevant in virtualization environment (where virtual disks smaller than 10 GiB are not uncommon) or certain architectures with special storage devices like s390 and ARM.

The final decision was to never disable snapshots automatically in the case of openSUSE, so the user will be forced to manually go through the Guided Setup and explicitly disable snapshots to install in a small disk. In the SLE case, it was decided to keep the traditional behavior (automatically disabling snapshots if really needed) but making the situation more visible by adding a previous sentence to explain how the initial proposal was calculated.

So the installation in a normal disk would look like this.

While the installation in a very small disk displays some information similar to the following screen (the wording was slightly improved after taking the screenshots).

The explanatory text preceding the list of actions will be available in all products based on SLE15, but will not be there for Leap 15.0, since the modification to the installer was not ready on time for the deadline and, moreover, would have been impossible to get the translations on time.

By the way, if you are interested in a more in-depth explanation on how the partitioning proposal adapts to all kind of situations like small disks and other scenarios, don’t hesitate to check Iván’s presentation at openSUSE Conference 2018 detailing its internals.

More parameter passing for s390

And talking about uncommon scenarios and the s390 architecture, you may remember that in the latest sprint we improved the handling of the persistent network device names kernel parameter for such systems. Shortly after, we found out a similar improvement was needed also for the FIPS parameter.

FIPS is a military encryption standard in USA. If the installation is started using the corresponding parameter, YaST will enforce strong encryption and will install an specific FIPS pattern. Moreover, after the recent fix, a system installed in hardened mode s390 will continue operating in this mode after the installation.

Fun with MD RAIDs

As SLE15 comes closer, future users start testing the system with more exotic and complex hardware setups. Same applies to openSUSE Leap 15.0 right after the official release. As a result of all that testing, we found several scenarios in which Storage-ng got confused about MD RAIDs defined by some specific hardware or manually by the user before starting the installation.

By default, the old storage didn’t handle partitions within software RAIDs and it didn’t handle software RAIDs directly on top of full disks (with no partitions in the physical disks). For the first version of Storage-ng present in Leap 15.0 and SLE15, we tried to implement the same behavior with the intention to rethink the whole thing and open new possibilities in the close future. Check more about the present and future of Storage-ng in Ancor’s talk at openSUSE Conference 2018.

Unfortunately, while trying to replicate the old storage behavior with software-defined MD RAIDs, we overlooked some heuristic that was hidden in the old implementation to recognize some special setups in which a given RAID device currently detected as regular software-defined RAIDs should be treated like hardware RAIDs. That’s the case of Software RAID Virtual Disks defined on a S130/S140 controller on DellEMC PowerEdge Servers through the BIOS Interface. We also found that some users used to produce a similar situation by manually creating software MD RAIDs and creating partitions within them before starting the installation.

With the preparation of SLE15 already in the final stages and with openSUSE Leap 15 already out, it was too late to introduce drastic changes in how MD RAIDs are detected and used. To mitigate the problem while limiting the potential breakage, we reintroduced an ancient installer parameter. Now, when we run the installer using LIBSTORAGE_MDPART=1, all existing software-defined RAIDs will be considered as BIOS RAIDs.

The new parameter is not available in Leap 15.0 (we added it too late) and will hopefully not be necessary anymore in future versions of SLE and openSUSE, since the short term plan is to redesign everything about how MD RAIDs are handled during installation.

And even more fun with MD RAIDs

Another example of RAID that looks like defined by software but is indeed assembled by BIOS is the Intel RSTe technology. In this case, the usage of LIBSTORAGE_MDPART is not needed, but still we found the bootloader installation to be broken because YaST was once again getting confused by the mixed RAID setup.

Fortunately it was possible to fix the issue and verify the solution in only two days, despite the YaST Team not having direct access to the hardware, thanks to the outstanding help of the user reporting the bug. Connecting users and developers directly always produces great results… and that’s one of the reasons open source rocks so much!

Improved error reporting for wrong bootloader in AutoYaST

That was not the only improvement in the bootloader handling done during this sprint. We also invested some time improving the user experience in AutoYAST, since the error message displayed when using an EFI variant not supported in the system architecture was far from being useful or even informative.

So alongside a more clear message, AutoYaST will now list all the possible values supported on the given architecture to better guide the user.

Setting the default subvolume name in AutoYaST

AutoYaST also received improvements in other areas, like making use of the new possibilities offered by Storage-ng. The new storage layer allows the user to set different default subvolumes (or none at all) for every Btrfs file system. As shown in the example below, a prefix name can be specified for each partition using the subvolumes_prefix.

As a consequence of the new behaviour, the old btrfs_set_default_subvolume_name tag is not needed and, therefore, it is not supported in Leap 15.0 and SLE15.

Skipping Btrfs subvolume creation

And more changes in AutoYaST that arrived just in time for SLE15 and openSUSE Leap 15.0. Recently, we have introduced a new flag in AutoYaST partition sections to skip the creation of Btrfs subvolumes because, due to a known limitation of our XML parser, it is not possible to specify an empty list.

So from now on, setting create_subvolumes to false will prevent AutoYaST from creating any Btrfs subvolumes in a given partition.

Keep it rolling!

As usual, the content of this post is just a small part of everything we did during the sprint. There were also many other fixes and improvements, from auto-repairing wrong partition tables (with different sizes than the underlying disk) during installation to better interaction with other components like udisk or mdadm auto-assembling and many other things in between.

But it’s time to go back to work and start implementing all the new ideas that emerged from the YaST Team Workshop and the openSUSE Conference. See you in the next report!

LEAP/SLE 15 is getting more stable and closer to be released, but to keep this process flowing, our team of bug killers is having a lot of work to do!

This last sprint we had several fixes for really special scenarios. The kind of problems that you can find once most of things are working fine! So let’s take a look at some of these cases and how we’re working to stabilize this upcoming release.

Keep predictable network devices settings on S390x

A not well-known feature in YaST is that many specific boot parameters for installation are also used on the target system. However, this approach has one exception: the S390 mainframe. In this case, many installation specific parameters should not go to the target system and, therefore, we ignore all installation parameter for this system.
In the last sprint, we worked on a bug, which reported that at least systemd predictable names for network devices should be also used for S390 systems, otherwise the configuration done during the installation won’t be valid in a running system as network names will differ. So, from SLE15 we start to keep the settings of predictable network names for S390 systems.

Fun with console configuration of GRUB2

Another report that helped us to learn about other not well-known features was the one reporting that bootloader module does not support multiple console outputs of GRUB2. After digging into some code, we found out that YaST bootloader takes only native terminal, gfxterm or serial console in consideration, but not a mix of them.
Once we looked at the manual of GRUB2, we learned that it supports some funny outputs such as morse code, PC beeper or simple data protocol using system speaker. Of course, YaST2 bootloader does not support all these options and when it gets one of them, it is treated as an unexpected value and bootloader fails.

As we are really close to Leap/SLE 15 release, we want to avoid big changes in the system. So we decided to handle this issue by showing a popup, which informs that the configuration contains an unexpected value and asks if the whole proposed configuration should be proposed again or if YaST should quit and let the user edit it manually.

If you are curious about how it looks like:

For Leap 15.1 / SLE 15 SP1 we plan to extend the values that we support to provide a nicer experience for the user.

Bootloader configuration during upgrade

Another reported issue in bootloader was also solved this last sprint. When upgrading from Leap 42/SLE 12 to Leap/SLE 15, if the user clicks on booting on the proposal, the system crashes. The reason is that usually on openSUSE 13.2/SLE 11 it needs to repropose bootloader during the upgrade. This is no longer required for the latest upgrade and, therefore, YaST does not expect that the user will click on it. We would like to remove this option completely, but YaST still support upgrade from SLE11 to SLE15, so we still need it there. In the end, the solution is to show a popup informing the user that the modification of bootloader is not supported during upgrade.

In short, take a look at the screenshot:

Fixing kdump on Xen

We got a report about kdump breaking when it is used in Xen. To explain the problem, we need to go back to how we configure a parameter and the reasons we implemented it in this way: In current versions (before this fix) when a user wants to use kdump, we configure the crashkernel kernel parameter for all targets, for the common kernel, Xen PV0 domain and also Xen guests. This approach worked very well in the past because traditional xenlinux ignores crashkernel for PV0 and just pass it to Xen VMs. However, the current pvops implementation of Xen no longer ignores this parameter and consequently, it results in breaking the Xen virtualization. The solution for this issue was pretty simple: YaST stopped to propose using crashkernel for Xen PV0 and everything works again. This is a perfect example to show how to understand the issue, sometimes, takes much more time than to fix it!

Improving the upgrade from SLE11/12 to SLE15

We are still improving the migration from SLE11 or SLE12 to SLE15 and this sprint we were focused on the automated registration upgrade using AutoYaST. If you are not familiar with the autoupgrade feature you can find more details in the documentation.

We already supported manual registration upgrade from the old products, but the automated way was still not adapted to the new SUSE Customer Center (SCC) API and it did not work correctly. This sprint we have adapted the registration upgrade code to correctly work in the interactive mode and also in the automatic upgrade.

The code was adapted to skip the user interaction and do everything manually when running in the autoupgrade mode. The only problematic part was how to handle multiple migration targets, which in the interactive upgrade we ask the user to choose one. To have a simple solution we decided to take the first migration, later (SP1) we might allow configuring this as well. But as now there is only one migration possible anyway, this looks like a good enough solution.

Falling back to the guided proposal

During this sprint, we were informed that AutoYaST was unable to display a proper error message when no partitioning section was specified and there was not enough disk space. The bug was rather easy to solve, but we wanted to take the opportunity to highlight how AutoYaST works when the partitioning section is missing from the profile.

In the past, AutoYaST implemented its own logic, different from the one used during a normal installation. Fortunately, as part of the adaptation to the new storage layer, AutoYaST relies now on the same code than the regular installation in order to propose a partitioning layout when the partitioning section is not specified. What is more, you can override some values which are defined in the product’s control file by setting them in the general/storage section from the AutoYaST profile.

Leap /SLE 15 is closed, but Tumbleweed is still rolling

We are really close to release Leap/SLE 15 and we are more focused on minimal changes that fix only critical stuff. On the other hand, Tumbleweed users are looking always for the latest and greatest features. In order to satisfy both groups, YaST separated Leap 15.1/SLE 15 and Tumbleweed in two different git branches. In this way, we can easily start adding new features, bug fixes and other improvements for Tumbleweed while we keep SLE15 stabilized. Besides that, there is another planned Service Pack for SLE12, and once we start to work at it, we’ll also create a new separated branch to include these changes. We also adapted all related infrastructure around the branches, such as CI, docker testing images, among others.

This way, we are able to allow you to enjoy the stable Leap 15 or the latest and hottest Tumbleweed.

Conclusion

We’re now working on our last sprint before the openSUSE Conference 2018. In two weeks we’ll come back with the highlights of Sprint 57 and until there we hope that you have already everything planned to enjoy the conference that will occur in Prague this year (we hope that you enjoy the city too). We’re looking forward to seeing you there!