Archive for the ‘GSOC’ Category

We are back to this blog after another three weeks of (mainly) bug-fixing. In the previous post we promised some news about the self-update functionality and about the LVM support in the new storage stack. We have that… and much more!

So this will be a long post, but it also hides some gems. You will have to keep reading in order to find them.

Self-update improvements

We have already mentioned in several previous reports the new self-update feature in YaST, which allows updating the installer itself before performing installation of the system.

But it turned out that the initial implementation had an important drawback. The self-update process happened after having performed some of the installation steps. Then, after updating the installer it was restarted and several of those steps lost their configuration or simply did their operations twice.

After some discussions we decided to move the self-update step earlier, at the very beginning. For downloading the updates we basically need just working network connection and initialized package management. So we moved the self-update step after the initial automatic network setup (DHCP) and added package initialization to the self-update step.

As you can see the self-update step is the very first step in the installation workflow, the language selection and the EULA dialog is displayed after the self update is finished and YaST is restarted. That means all the following steps do not need to remember their state as they will not be called twice after the restart.

The disadvantage is that we had to drop some features. The self-update step happens before the language selection and the optional disk activation. That means by default the self-update progress (and potential error messages) will be displayed in English. But you can still use the “language” boot option and set the language manually via linuxrc.

On the bright side, we fixed like half a dozen of reported bugs just by relocating the self-update process. So we are pretty sure it’s worth the price.

Gem one: using the info boot parameter

The info boot parameter is a pretty old linuxrc option but it is probably not known well. The parameter is an URL which points to a text file which can contain more boot options.

When we tested the updated self-update described above we needed to build a driver update disk and pass several boot options. To avoid repeating the same options on the boot command line and to share the boot options across the team we created an info.txt file with content like

insecure=1
startshell=1
dud=ftp://example.com/self_update.dud

Then you simply boot the installation with info=ftp://example.com/info.txt and linuxrc will read the additional parameters from the file. This can save you a lot of typing, especially if you need to repeat the tests many times.

Fixed a security bug for 7 (yes, seven) different SLE releases

Some weeks ago, during a routine code review, our security experts found a vulnerability in YaST’s libstorage related to the way we provide the encryption passwords to some external commands. It is debatable how dangerous this threat really is. It was never a problem during system installation, but it would affect admins who create encrypted partitions (mostly encrypted LVM physical volumes) or crypto files in the installed system.

A potential attacker with access to /tmp could intercept the password in the very precise moment in which the “cryptsetup” or “losetup” command are invoked by YaST. It’s really only a matter of milliseconds. But we don’t want to take any risks, however small they may be.

So not only did we fix that for the current code streams, we backported it to all the SLE releases out there that are still supported (even though in some cases it’s just a single customer) – back to SLES-10 SP3 from late 2009. That meant backporting the fix to no less than 7 SLE releases (for Leap, those fixes are picked automatically).

As you can imagine, this got more difficult the farther back in history we went: In a central library like libstorage, things are constantly changing because the tools and environment (kernel, udev, you name it) are constantly changing. There was only a single case where the patch applied cleanly; in all other cases, it involved massive manual work (including testing, of course).

Was this fun? No, it certainly was not. It was a tedious and most frustrating experience. Do we owe it to our users (paying customers as well as community users) to fix security problems, however theoretical they are? Yes, of course. That’s why we do those things.

Storage reimplementation: every LVM piece in its place

As time permits, we keep adding new features to the future libstorage replacement. During previous sprints we added support to read and manipulate all kinds of LVM block devices (PVs, VGs and LVs) but an important aspect was missing: deciding the order of the operations is as important as performing them. We instructed the library about the dependencies between operations and implemented several automated test cases to ensure we don’t try to do not-so-smart things like removing a physical volume from a volume group and shrinking its logical volumes afterwards.

The good thing about our automated test-cases is that they generate nice graph that are quite useful to illustrate blog posts. 🙂

Gem two: enjoy Google Summer of Code result

As you may know, openSUSE is one of the Free Software organizations selected to take part in Google Summer of Code 2016. For YaST that means we had the huge pleasure of having Joaquín Yeray as student. You can know more about him and his experience diving into YaST and Open Source in his GSoC blog.

But the openSUSE community is not only gaining a new member, we also have a new YaST module. The yast2-alternatives package has already been accepted into Tumbleweed and will be also part of Leap 42.2. So we have a new gadget in our beloved configuration Swiss Army knife!

We liked Joaquín and his module so much that we are revamping the YaST development tutorial to be based on his module (instead of yast2-journal). He is already working on that, so hopefully we will have Joaquín around quite some time still. 😉

Unify license handling screens

We got a report about the license agreement screen in automatic installation (AutoYaST) being different to the one showed during common installation. So we decided to take a look to the problem and unify them. We are in a quite late phase of the development process of both the next SLE and the next Leap, so we decided to not unify the code but simply adapt one dialog to look like the other. Also we are after string freeze due to translations, so we had to use a trick and reuse another already translated text. We also took the opportunity to fix some small usability problems.

This is one of those cases in which some images are worth a thousand words, so in order to understand what we did, take a look at the description of this pull request, which includes many images (too many for this blog post).

Smarter check to avoid duplicated repositories

The openSUSE software server defines the online repositories which can be added during installation. The openSUSE DVD also specifies its own online repositories that are always added to the system. And these repositories overlap.

In openSUSE 42.1 it happened that one repository was added twice, even though there was already a check to avoid that. So we investigated why.

We found that the URLs for the problematic repository were not exactly the same, one of them had a trailing slash. Therefore we made the URL comparison more tolerant and if the URLs differ only by the trailing slash, they are still considered the same.

After the fix all repositories are added only once, without any duplicates.

Gem three: we are looking for new teammates!

After 12 sprint reports, most readers would have already realized that the life as a full-time YaST developer is everything but boring… and that we are always pretty busy. The fun and the work are better when you share them so… we are looking for a new hero to join us in our journey.

Even if you don’t feel hacking in YaST would be your thing, maybe you are interested in any of the other jobs at SUSE.

Improved documentation about YaST environment variables

The behaviour of YaST can be affected by several environment variables, but not all of them are well known by everybody. During this sprint we also decided to invest some time documenting them better. The resulting document will be soon properly integrated in our centralized documentation for developers, but you can sneak it already here.

Branching Tumbleweed and the upcoming stable releases

Most of the features and bug-fixes we have blogged about in the last months were incorporated to Tumbleweed, the upcoming Leap 42.2 and the future SLE 12-SP2, since we always try to keep those three codebases as close as possible to each other.

Now Leap 42.2 and SLE 12-SP2 are close enough to their release date, so we plan to be more conservative with the changes. At the end of this sprint we decided to branch the code for Tumbleweed and for the stable siblings. From now on, most exciting stuff will appear only in Tumbleweed, with SLE 12-SP2 and Leap 42.2 becoming more and more boring.

And the wheel keeps on turning

So that was a very minimal selection of the most interesting stuff from the just finished sprint. What comes next? Another sprint, of course! We have already planned some interesting stuff for it, like integrating the new partitioning proposal into the installer or finishing the ultra-cool UI designer that was started during latest Hack Week.

As always, you can follow development in a daily basis in the usual channels (#yast IRC channel and the yast-devel mailing list) or wait another three weeks for the next sprint report. Meanwhile… have a lot of fun!

as I already I wrote in an earlier post the last week I worked on implementing osc2’s “request”
command. In order to create a new submit request the user has to runosc request create --submit api://src_project/src_package api://tgt_project
What’s still needed is a way to specify a revision of the src_package. It isn’t possible
to introduce a “–revision ” option because it’s possible to specify multiple submit
actions for a request (so we can’t distinguish to which –submit action a revision
belongs to).
Finally I implemented darix’ cool idea: simply use a svn-like syntax:osc request create --submit api://src_project/src_package@rev api://tgt_project

here’s a small summary of the 12th (coding) week. The last week I continued
the work on the new user interface. As a result the “request” command is more
or less implemented. Yesterday I pushed the current code and it’s also possible
to play around with it:

Example:
In order to list a request it is sufficient to run
“python cli.py request api://your_project” (Note: all https requests are
_insecure_ – so far the certificate etc. is not checked).

Some implementation details:
In my last mail/post I described how a new (sub)command can be specified
so I’ll leave out the details here.
In the following I’ll shortly explain how the arguments specified by the
user at the commandline are passed to a function (which does the actual
work – like listing the requests).

As we can see the project and package parameters are optional. After
the parsing process a so called “info” object is returned which encapsulates
the specified arguments. Assume the user runs “request list api://test_prj”
then the info object has the following attributes:

The question is how can we pass this data to the request.list function?
A simple (and naive) approach would be to simply pass the “info” object
to the request.list function that is “list” has the following signature
“def list(info)”. As a consequence inside the method we always have to
use “info.project”, “info.package” etc. which is a bit awkward – at least
for parameters which are quite frequently used in the method definition.

So it would be nice if there’s a way to pass frequently used parameters
directly to the method (that is they’re declared as formal parameters
in the method definition) and infrequently used parameters can still be
accessed via the info object. Exactly like this it is currently
implementend in osc2.

here’s a small summary of the 11th (coding) week. Last week I worked on
implementing the new commandline interface. While doing so I faced several
“issues”:

How to combine argparse and our oscargs url-like syntax?
Basically we have to run our oscargs parser on the result which is
returned by argparse.ArgumentParser’s parse_args method. The problem is
that both parsers have a different “syntax” that is using a naive approach
will lead to some redundancies (we specify the ui twice: one time for
argparse and one time for oscargs). In order to avoid this we need some
“interface” to which the oscargs syntax is passed and which configures
the argparse parser accordingly.

How to support custom commands?
We also have to provide an “easy” way to specify custom commands.
Additionally it might be handy if existing commands can be enhanced
(either by adding additional options etc. or by adding a new subcommand
etc.). The best would be if the user simply drop his/her plugins in a
specific directory and osc will scan this directory and use the new
plugins/commands.

Specifying the ui programmatically is somehow confusing/cluttered. It would
be much better if the ui can be specified in a more “declarative” way
without the syntactic “overhead” (well that’s a bit exaggerated) which
is needed to configure the parser. Additionally it would be nice to have
a convenient way to specify a multi line description for a command
(hardcoding the str into the source makes the code “harder” to read).

Finally I ended up with a small DSL which can be used to specify the
ui in a “declarative” way (the initial idea + syntax is taken from the
django framework (see [1])).

Example:
Assume we want to implement a request command which consists (for the
sake of simplicity) of 2 subcommands “list” and “accept”. This can be
specified like the following:

How does it work?
First of all each class which defines a command or subcommand has to inherit
from class “CommandDescription”. If a subcommand is to be defined it also
has to inherit from the “parent” command (that is in our example “RequestList”
and “RequestAccept” inherit from class “Request” (which in turn inherits from
class “OscCommand” (from this class all toplevel commands have to inherit))).
In short: with the help of the inheritance hierarchy it is possible to define
a command <- subcommand hierarchy.

Note: actually the classes "RequestList" and "RequestAccept" only inherit
from "CommandDescription". The "parent" command base class is only needed
for a "marking" purpose (it is filtered out with the help of a metaclass
when the concrete class is "constructed" – I'll leave out the details for
now and may write a dedicated blogpost about it).

Now the remaining task is to define and implement the commands (note: we will
definitely not finish the project on the "suggested pencils down" date and
use the week until the "firm pencils down" date for coding…).

here’s a small summary of the 9th (coding) week. Last week I worked
on the fetcher and cache manager code. In order to support all
features some of the existing classes had to be enhanced with some
more parameters.

Done:

cache manager code

BinaryList class supports view=cpio

RORemoteFile class supports lazy_open
parameter (by default the file is opened lazily that is when a
read request is issued) (for the fetcher code we use
lazy_open=False)

minor changes in the httprequest module (AbstractHTTPRequest
supports the same query parameter more than once)

The fetcher code is more or less done (not yet committed) and will be
finished by friday evening (I’ve some exams this week…).

here’s a small summary of the 8th (coding) week. The last days I
was working on getting build support into the osc2 library. I just
checked in a BuildInfo and Builder class (+ testcases). Here’s a
brief example how they can be used:
# example how to use the Builder class
builder = Builder(su_cmd=Builder.SUDO, root='/var/tmp/build-root')
builder.rpmlist = '/path/to/rpmlist'
builder.dist = '/path/to/buildconfig'
builder.without = 'feature1'
builder.without += 'feature2'
builder.arch = 'x86_64'
# run the build:
builder.run('/path/to/osc.spec')
# this executes:
sudo /usr/bin/build --arch x86_64 --dist /path/to/buildconfig \
--root /var/tmp/build-root --rpmlist /path/to/rpmlist \
--without feature1 --without feature2 /path/to/osc.spec

Basically Builder is just a wrapper around the build script
(all options will be passed to the build script; if an option contains
a “-” character like “vm-type” it can be set like this:
“binfo.vm_type = ‘xen'” (a “_” will be replaced with “-“)).

here’s a small summary of the 7th (coding) week. Last week
I was really busy with university stuff (it was the second
last week in the lecture period so I had to recap quite
some stuff) and didn’t manage to work much on the GSoC project.
I’m going work off the TODO this weekend.

here’s a small summary of the 6th (coding) week. Last week I
continued working on the build module and developed a concept
for the package fetcher (and discussed some parts
with darix:) ).
The main idea is to modularize the fetcher code, verify code etc.
Thus we have a fetcher class which takes care of retrieving the
packages. The fetcher class utilizes a “cache manager” which takes
care of storing the fetched packages on the filesystem. The goal is
that at some point in time one can exchange the “simple” cache manager
with a more “clever” cache manager (which for instance cleans up the
cache from time to time or only allows exactly one version of a
package in the cache etc.). In order to achieve this no code in the
fetcher has to be touched – instead it’s sufficient to pass a different
cache manager object to the fetcher.
Additionally the fetcher provides some hooks like pre, pre_fetch,
post_fetch and post. For instance a post_fetch hook can be used to
verify the just fetched package etc.

here’s a small summary of the 5th (coding) week. Last week I spent
most of my time with working on the cpio module. Finally I ended
up with a complete rewrite of osc’s “old” cpio module. The
cpio implementation details are taken from the cpio 2.11 package.
Currently only the “new ascii” format is supported and we can
only handle regular files (this is sufficient for our needs). Here’s
a small example how to use it:
# cpio_open is just a convenience method for CpioArchive(filename=fname)
with cpio_open(fname) as archive:
for archive_file in archive:
# print filename
print archive_file.hdr.name
# print contents
print archive_file.read()

It’s also possible to pass a file-like object to a CpioArchive
instance (in this case we do not need to use the with statement).
Also it can be “easily” enhanced to support different cpio formats
(one just have to write the specific ArchiveReader and ArchiveWriter
classes:) ). The code is available at [1] and the testcases at [2].

Todo for this week:
* finish the build module
* start with working on the package fetching code

here’s a small summary of the 3rd and 4th (coding) week . First of all I
wasn’t able to do lots of work in week 3 and 4 so I’m still working on
the new build module for the osc2 library.
The initial plan was to copy/reuse some of the existing modules from the
(old) “osc” like the cpio [1] and packagequery modules. But I decided to
refactor/rewrite cpio module for the following reasons:

“save” disc space:
In our scenario we retrieve a cpio archive from the api (which contains
binary packages for example). The old cpio module expects a filename
in order to “unpack” the archive – that is the file has to be stored on
disc first. Consequently approximately 2 * of free disc
space is needed.
The new idea is that we pass a file-like object (in our case an object
which inherits from “AbstractHTTPResponse”) to the cpio module and
unpack the archive “on the fly” (without storing the http response
on disc first).

have testcases:
The old cpio module has no testcases (because some time ago I didn’t
follow the TDD approach;) ). For nearly all modules in osc2 there exist
testcases (white-box tests) thus it would be nice if we have some
testcases for the cpio module, too (theoretically we could add some
black-box tests (the current methods aren’t really testable thus
white-box tests aren’t possible)).

have a nice pythonic interface:
The new interface will look like this:
from cpio import cpio_open
# let "f" be a file-like object (for instance a http response)
with cpio_open(fobj=f) as cpio_archive:
for a_file in cpio_archive:
# store file (with correct permissions etc.) in os.curdir
a_file.write(os.curdir)
# alternatively it's also possible to read some data (instead of
# writing it to disc) via a_file.read(len)

We will also support a plain filename in cpio_open.

Currently the cpio module will only support the “new ascii” (ascii SVR4
no CRC) format and regular files (that’s sufficient for our needs). But
it will be possible to simply pass in a class for a different format
(that is no code has to be altered in order to support a new format).

Finally this will be finished by the end of this week.
If you have any questions or suggestions please tell me:)