Posts Tagged ‘Infrastructure’

Before the dawn of 2015 and as announced at oSC14 work has already started for yet another awesome openSUSE Conference. For 2015 the city of The Hague in theNetherlands will host our Annual Conference. This year the openSUSE community is gearing up and is ready to accept people from everywhere to a 4 day conference where there will be a lot of Learning and Hacking and Partying. The organizing team needs YOUR HELP to make this happen. So starting today you can sign up to join forces!

Whatever your skill level you are welcome to participate. We are looking for people willing to help out with A/V, Design, Networking, Promotion, Software, finding Speakers and helping out with the Venue itself. Join us for an awesome ride, with a fantastic destination. Make it your own event!

We use the openSUSE progresstracker in this project to keep track of the tasks. The documents are used to suggest announcements. Every two weeks we have project meetings on the opensuse-project channel on freenode.

Anyone is welcome to participate! To do so:

Sign up for an openSUSE account and sign into this tool
Assign an existing issue to yourself or create an issue with a task

If you need help with this tool (Redmine) have a look at the user guide .

RSYNC and rsync.opensuse.org

But we faced an issue with the automation that creates the content of the “hotstuff” rsync modules: normally a script analyzes the log files of download.opensuse.org and arranges the content of these special rsync modules to provide always the most requested files, so our users have a good chance to find a very close mirror for their packages. But currently the script is not producing what we expect: it empties all those hotstuff modules. As the core developer behind this script comes back from vacation on Monday, we hope he can quickly fix the problem. For now we disabled the “hotstuff” modules (means on rsync.opensuse.org: we disabled rsync completely for now) to avoid problems.

If you want to sync packages to your local machine(s) via rsync: please pick a mirror from our page at mirrors.opensuse.org providing public rsync.

New hardware

All the racks of the OBS reference server

You may have noticed already that the openSUSE team installed a new version of openQA on the production server. An additional news item might be that this new version has seen also new hardware to run faster than ever.

But not only openQA, also the database cluster behind download.opensuse.org has seen a hardware upgrade. The new servers allow to run the database servers as virtual machines, able to have the whole database structure stored in RAM (you hopefully benefit from the faster response times on download.opensuse.org already). And the servers still have enough capacity left, so we can now also visualize the web servers providing the download.opensuse.org interface. We are currently thinking about the detailed setup of the new download.opensuse.org system (maybe using ha-proxy here again? maybe running mirrorbrain in the “no local storage” mode? …) – so this migration might take some more time, but we want to provide the best possible solution to you.

Admins on openSUSE Conference

These year, three of our main European openSUSE administrators are able to attend to the openSUSE Conference in Dubrovnik:

Markus RÃ¼ckert

Martin Caj

Robert Wawrig

And they will not only participate: instead they are providing talks and help with the infrastructure and video recording of the venue. So whenever you see them: time to spend them a drink or two :-)

After the outage 1 month ago, it seems rsync.opensuse.org has similar hardware problems again.

Again we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

[Update]: the problem seems to be a broken hard disk – and a hardware controller who can not really handle this degraded RAID array. For the moment, everything is up and running again, but we are now actively searching for replacement hardware…

Looks like the hardware behind rsync.opensuse.org now finally reaches it “end of life” status: we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

In the Service Teamâ€™s Cave, where the infrastructure of openSUSE servers and services runs, the openSUSE Service Team faced an issue: requests to admin@opensuse.org were managed only by mail, making it hard to keep track of all the open issues and leading to coordination problems. As some requests to this list also contain log in credentials, the list itself could not have a public archive. This could have exposed sensitive data to the public. So it is always complicated telling people what’s going on there, and even more complicated, allowing interested people to subscribe. (Please note: including credentials in plain Emails is never ever a good idea – it is even not the intention of the Service Team to get such credentials. But sometimes people don’t care about their sensitive data, or just realize too late that their log files contain information that should not be visible).

But openSUSE – and especially the administration of all openSUSE services – is all about collaboration and communication. So hiding in a small cave might not be a good idea if you want to get some helping hands or reach out for collaboration.

Today we took one big step forward with our infrastructure by integrating admin@opensuse.org into the ticket system available at http://progress.opensuse.org/ ! At first this may not sound very interesting, but please remember that this service is already integrated into our authentication infrastructure. Now everyone with an openSUSE account is able to Â check the state of public tickets (warning: tickets are set to private per default), create new ones a have a look at other public modules of this “openSUSE admin”-project – or become part of the team.

Just to avoid confusion: sending an email to admin@opensuse.org is not only still possible but also the preferred way to reach us.

For coordination and to be “reachable” for all those guys hanging around at some IRC channel, we now also have a public channel on irc.freenode.net: #opensuse-admin Feel free to say hello, thank you, or ask us questions.

On Saturday, December 1st 2012, at approximately 05:00 UTC our data center team will do a backend storage upgrade. The planned window is 8-10 hours for the maintenance, and specific applications will probably be not available until 24:00 UTC as listed below.

Our public rsync server ‘rsync.opensuse.org’ will be not available on Monday, 2012-09-17, starting 11:00 local time (CEST) which is 17th of September, 09:00 UTC. The downtime is expected to last for three hours.

The SAN array of the backend server server lost 3 hard disks over the weekend.

That means the array with the built RPMs was broken. We checked and replaced a lot of files from backups – but since not all binary parts of the projects are in backup we need to rebuild some of them (31 from 24,194) afterwards.

The good news: sources and project configurations were affected by this.

Update 2011-03-28 13:44 UTC: The login system is a bit unstable at the moment and might be occasionally unavailable for a few minutes again, they will be back soon. We’re working on getting everything running smoothly again.