gentoo – Rich0's Gentoo Bloghttps://rich0gentoo.wordpress.com
Fri, 18 Aug 2017 02:52:46 +0000enhourly1http://wordpress.com/https://s2.wp.com/i/buttonw-com.pnggentoo – Rich0's Gentoo Bloghttps://rich0gentoo.wordpress.com
Gentoo Ought to be About Choicehttps://rich0gentoo.wordpress.com/2016/02/27/gentoo-ought-to-be-about-choice/
https://rich0gentoo.wordpress.com/2016/02/27/gentoo-ought-to-be-about-choice/#commentsSun, 28 Feb 2016 02:07:55 +0000http://rich0gentoo.wordpress.com/?p=239]]>“Gentoo is about choice.” We’ve said it so often that it seems like we just don’t bother to say it any more. However, with some of the recent conflicts on the lists (which I’ve contributed to) and indeed across the FOSS community at large, I think this is a message that is worth repeating…

Ok, bare with me because I’m going to talk about systemd. This post isn’t really about systemd, but it would probably not be nearly as important in its absence. So, we need to talk about why I’m bringing this up.

How we got here

Systemd has brought a wave of change in the Linux community, and most of the popular distros have decided to adopt it. This has created a bit of a vacuum for those who strongly prefer to avoid it, and many of these have adopted Gentoo (the only other large-ish option is Slackware), and indeed some have begun to contribute back. The resulting shift in demographics have caused tensions in the community, and I believe this has created a tendency for us to focus too much on what makes us different.

Where we are now

Every distro has a niche of some kind – a mission that gives it a purpose for existence. It is the thing that its community coalesces around. When a distro loses this sense of purpose, it will die or fork, whether by the forces of lost contributors or lost profits. This purpose can certainly evolve over time, but ultimately it is this purpose which holds everything together.

For many years in Gentoo our purpose has been about providing choices, and enabling the user. Sometimes we enable them to shoot their own feet, and we often enable them to break things in ways that our developers would prefer not to troubleshoot. We tend to view the act of suppressing choices as contrary to our values, even if we don’t always have the manpower to support every choice that can possibly exist.

The result of this philosophy is what we all see around us. Gentoo is a distro that can be used to build the most popular desktop linux-based operating system (ChromeOS), and which reportedly is also used as the basis of servers that run NASDAQ[1]. It shouldn’t be surprising that Gentoo works with no fewer than 7 device-manager implementations and 4 service managers.

Still, many in the Linux community struggle to understand us. They mistake our commitment to providing a choice as some kind of endorsement of that choice. Gentoo isn’t about picking winners. We’re not an anti-systemd distro, even if many who dislike systemd may be found among us and it is straightforward to install Gentoo without “systemd” appearing anywhere in the filesystem. We’re not a pro-systemd distro, even if (IMHO) we offer one of the best and undiluted systemd experiences around. We’re a distro where developers and users with a diverse set of interests come together to contribute using a set of tools that makes it practical for each of us to reach in and pull out the system that we want to have.

Where we need to be

Ultimately, I think a healthy Gentoo is one which allows us all to express our preferences and exchange our knowledge, but where in the end we all get behind a shared goal of empowering our users to make the decisions. There will always be conflict when we need to pick a default, but we must view defaults as conveniences and not endorsements. Our defaults must be reasonably well-supported, but not litmus tests against which packages and maintainers are judged. And, in the end, we all benefit when we are exposed to those who disagree and are able to glean from them the insights that we might have otherwise missed on our own.

When we stop making Gentoo about a choice, and start making it about having a choice, we find our way.

Filed under: foss, gentoo, linux, Uncategorized]]>https://rich0gentoo.wordpress.com/2016/02/27/gentoo-ought-to-be-about-choice/feed/18rich0Running cron jobs as units automaticallyhttps://rich0gentoo.wordpress.com/2015/07/18/running-cron-jobs-as-units-automatically/
https://rich0gentoo.wordpress.com/2015/07/18/running-cron-jobs-as-units-automatically/#commentsSat, 18 Jul 2015 16:00:42 +0000http://rich0gentoo.wordpress.com/?p=237]]>I just added sys-process/systemd-cron to the Gentoo repository. Until now I’ve been running it from my overlay and getting it into the tree was overdue. I’ve found it to be an incredibly useful tool.

All it does is install a set of unit files and a crontab generator. The unit files (best used by starting/enabling cron.target) will run jobs from /etc/cron.* at the appropriate times. The generator can parse /etc/crontab and create timer units for every line dynamically.

Note that the default Gentoo install runs the /etc/cron.* jobs from /etc/crontab, so if you aren’t careful you might end up running them twice. The simplest solutions this are to either remove those lines from /etc/crontab, or install systemd-cron using USE=etc-crontab-systemd which will have the generator ignore /etc/crontab and instead look for /etc/crontab-systemd where you can install jobs you’d like to run using systemd.

The generator works like you’d expect it to – if you edit the crontab file the units will automatically be created/destroyed dynamically.

One warning about timer units compared to cron jobs is that the jobs are run as services, which means that when the main process dies all its children will be killed. If you have anything in /etc/cron.* which forks you’ll need to have the main script wait at the end.

On the topic of race conditions, each cron.* directory and each /etc/crontab line will create a separate unit. Those units will all run in parallel (to the extent that one is still running when the next starts), but within a cron.* directory the scripts will run in series. That may be a bit different from some cron implementations which may limit the number of simultaneous jobs globally.

All the usual timer unit logic applies. stdout goes to the journal, systemctl list-timers shows what is scheduled, etc.

Filed under: gentoo, linux, systemd]]>https://rich0gentoo.wordpress.com/2015/07/18/running-cron-jobs-as-units-automatically/feed/9rich0Quick systemd-nspawn guidehttps://rich0gentoo.wordpress.com/2014/07/14/quick-systemd-nspawn-guide/
https://rich0gentoo.wordpress.com/2014/07/14/quick-systemd-nspawn-guide/#commentsMon, 14 Jul 2014 20:31:47 +0000http://rich0gentoo.wordpress.com/?p=229]]>I switched to using systemd-nspawn in place of chroot and wanted to give a quick guide to using it. The short version is that I’d strongly recommend that anybody running systemd that uses chroot switch over – there really are no downsides as long as your kernel is properly configured.

Chroot should be no stranger to anybody who works on distros, and I suspect that the majority of Gentoo users have need for it from time to time.

The Challenges of chroot

For most interactive uses it isn’t sufficient to just run chroot. Usually you need to mount /proc, /sys, and bind mount /dev so that you don’t have issues like missing ptys, etc. If you use tmpfs you might also want to mount the new tmp, var/tmp as tmpfs. Then you might want to make other bind mounts into the chroot. None of this is particularly difficult, but you usually end up writing a small script to manage it.

Now, I routinely do full backups, and usually that involves excluding stuff like tmp dirs, and anything resembling a bind mount. When I set up a new chroot that means updating my backup config, which I usually forget to do since most of the time the chroot mounts aren’t running anyway. Then when I do leave it mounted overnight I end up with backups consuming lots of extra space (bind mounts of large trees).

Finally, systemd now by default handles bind mounts a little differently when they contain other mount points (such as when using –rbind). Apparently unmounting something in the bind mount will cause systemd to unmount the corresponding directory on the other side of the bind. Imagine my surprise when I unmounted my chroot bind to /dev and discovered /dev/pts and /dev/shm no longer mounted on the host. It looks like there are ways to change that, but this isn’t the point of my post (it just spurred me to find another way).

Systemd-nspawn’s Advantages

Systemd-nspawn is a tool that launches a container, and it can operate just like chroot in its simplest form. By default it automatically sets up most of the overhead like /dev, /tmp, etc. With a few options it can also set up other bind mounts as well. When the container exits all the mounts are cleaned up.

From the outside of the container nothing appears different when the container is running. In fact, you could spawn 5 different systemd-nspawn container instances from the same chroot and they wouldn’t have any interaction except via the filesystem (and that excludes /dev, /tmp, and so on – only changes in /usr, /etc will propagate across). Your backup won’t see the bind mounts, or tmpfs, or anything else mounted within the container.

The container also has all those other nifty container benefits like containment – a killall inside the container won’t touch anything outside, and so on. The security isn’t airtight – the intent is to prevent accidental mistakes.

Then, if you use a compatible sysvinit (which includes systemd, and I think recent versions of openrc), you can actually boot the container, which drops you to a getty inside. That means you can use fstab to do additional mounts inside the container, run daemons, and so on. You get almost all the benefits of virtualization for the cost of a chroot (no need to build a kernel, and so on). It is a bit odd to be running systemctl poweroff inside what looks just like a chroot, but it works.

Note that unless you do a bit more setup you will share the same network interface with the host, so no running sshd on the container if you have it on the host, etc. I won’t get into this but it shouldn’t be hard to run a separate network namespace and bind the interfaces so that the new instance can run dhcp.

How to do it

So, getting it actually working will likely be the shortest bit in this post.

You need support for namespaces and multiple devpts instances in your kernel:

That’s it – you can exit from it just like a chroot. From inside you can run mount and see that it has taken care of /dev and /tmp for you. The “.” is the path to the chroot, which I assume is the current directory. With nothing further it runs bash inside.

If you want to add some bind mounts it is easy:

systemd-nspawn -D . –bind /usr/portage

Now your /usr/portage is bound to your host, so no need to sync/etc. If you want to bind to a different destination add a “:dest” after the source, relative to the root of the chroot (so –bind foo is the same as –bind foo:foo).

If the container has a functional init that can handle being run inside, you can add a -b to boot it:

systemd-nspawn -D . –bind /usr/portage -b

Watch the init do its job. Shut down the container to exit.

Now, if that container is running systemd you can direct its journal to the host journal with -j:

systemd-nspawn -D . –bind /usr/portage -j -b

Now, nspawn registers the container so that it shows up in machinectl. That makes it easy to launch a new getty on it, or ssh to it (if it is running ssh – see my note above about network namespaces), or power it off from the host.

Filed under: foss, gentoo, linux]]>https://rich0gentoo.wordpress.com/2014/07/14/quick-systemd-nspawn-guide/feed/22rich0Quick EC2 Backups with Duplicityhttps://rich0gentoo.wordpress.com/2014/02/04/quick-ec2-backups-with-duplicity/
https://rich0gentoo.wordpress.com/2014/02/04/quick-ec2-backups-with-duplicity/#respondTue, 04 Feb 2014 22:07:23 +0000http://rich0gentoo.wordpress.com/?p=216]]>I’ve been doing online EC2 backups on my Gentoo box for a while, but switched to Duplicity a few months ago and have been very happy with the results. Setting this up took some trial and error, so I figured I’d share my config in case others find it useful. But first, here’s why I switched…

Duplicity is based on librsync, and is designed to be a simple command-line-based tool. Due to librsync the incremental backups it generates are VERY small – if you change one byte in a 2GB file it sends only a few bytes, just like rsync. It supports encryption and EC2 upload (and a bunch of other options) natively, which means I can ditch a bunch of shell scripting. It also stores encrypted manifests on the backup destination, which means that if your local index gets out of sync it can quickly synchronize and continue sending incrementals.

My configuration makes use of an alternates-filename patch which you can find in the duplicity bugzilla, or obtain from the rich0 Gentoo overlay. v0.6.24 of Duplicity will actually implement this in a slightly different manner, so if you do use this patch be aware that you’ll need to do new full backups after you upgrade. You can just drop the –alternate-filenames option and not use the patch, but if you do so you won’t be able to configure Amazon S3 to archive your difftar files to Glacier.

Once you install duplicity from either Gentoo portage or using my overlay, you’ll need your EC2 credentials. Then to execute a backup you can run:

You should substitute your AWS credentials. Setting TMPDIR is optional, but if you’re running tmpfs you might not have space for all the stuff you’re backing up. Setting archive-dir is also optional, but the archives are expendable and I don’t want it in my home getting backed up – if they get deleted duplicity will automatically fetch them back from EC2 (which will cost you money – so don’t delete them needlessly).

Exclude-if-present means that if you touch .noonlinebackup in a directory it won’t back up that directory. The exclude list is just a file with one path per row which will be excluded. You can have as many include options as you like. The exclude/include/exclude followed by a backup on / was the only way I could get it to back up paths relative to root while excluding files that were otherwise in the include paths. Otherwise the default include/exclude order wasn’t terribly helpful, but I might not be grokking the intent.

Full-if-older-than tells duplicity to create a new full backup every 60d. You can separately run a variation of this to do cleanup:

30GB of source data with 2GB of files that changed, and this was protected by transferring 36MB of data with the whole operation completed in 10 minutes.

By using alternative filenames you can set a prefix in S3 to archive your difftar files to glacier. I wouldn’t archive anything else – the manifests do not account for much space and if you lose your local copy it will re-download them (which is very expensive for glacier if not carefully managed). The future Duplicity release will allow you to specify separate prefixes for each file type created so that you can just put your difftars in a separate directory.

Filed under: foss, gentoo, linux]]>https://rich0gentoo.wordpress.com/2014/02/04/quick-ec2-backups-with-duplicity/feed/0rich0btrfs and snapper with portage on Gentoohttps://rich0gentoo.wordpress.com/2013/11/26/btrfs-and-snapper-with-portage-on-gentoo/
https://rich0gentoo.wordpress.com/2013/11/26/btrfs-and-snapper-with-portage-on-gentoo/#commentsTue, 26 Nov 2013 20:52:42 +0000http://rich0gentoo.wordpress.com/?p=208]]>This is just a quick share-a-recipe post to introduce snapper to anybody who hasn’t heard of it, and explain how to use it.

Snapper is a utility that manages btrfs snapshots. One of the nice features of btrfs is that snapshots are cheap (virtually instant, and consume space only as changes accumulate), and easy to access. Snapper allows you to automatically create and manage them based on time, events, manual action, etc.

Once snapper is set up you can display a list of snapshots. I have 10 hourly snapshots, 10 daily snapshots, and snapshots from before/after each emerge. I can diff them, browse them, etc. Btrfs snapshots can be browsed right from the filesystem, so if I nuke /etc/passwd I can always do a cp /.snapshots/1875/snapshot/etc/passwd /etc/passwd to restore one from a few hours before (though I do also have /etc in a git repo).

Snapper is currently available in the sunrise overlay – I won’t spend time on how to set that up/etc. Also, I’ve had time-based snapshots running for a while now and my memory is hazy as to whether I had to do anything to get those working – it just requires sticking some scripts in /etc/cron.*/ and creating a config file containing your policies.

What I did want to post is a recipe for getting pre/post-emerge snapshots working. All you need to do is add some lines to /etc/portage/bashrc:

Filed under: foss, gentoo, linux]]>https://rich0gentoo.wordpress.com/2013/11/26/btrfs-and-snapper-with-portage-on-gentoo/feed/4rich0The Balance of Power in Gentoohttps://rich0gentoo.wordpress.com/2013/11/17/the-balance-of-power-in-gentoo/
https://rich0gentoo.wordpress.com/2013/11/17/the-balance-of-power-in-gentoo/#commentsSun, 17 Nov 2013 16:39:37 +0000http://rich0gentoo.wordpress.com/?p=200]]>The recent concerns with the request to re-populate QA have re-opened a debate that is a few years old now. I’ve already made some specific recommendations on the lists, but I wanted to step back and explain why I feel the way I do.

Gentoo’s system of governance has some internal ironies – ones which occasionally even lead to calls to establish a benevolent dictator position. I think the mistake that Gentoo makes is that the problem is perceived as being democracy, when in reality the problem is with competing governance bodies with differing constituencies…

The greatest irony is that the same organization that contemplates appointing a supreme dictator often recoils from centralizing power in committees.

Please note that nothing I write here is really directed at any particular project/org/etc within Gentoo, or at any individuals who have served in governing roles in the past/present. My intent is to focus entirely on the system of governance that Gentoo uses, and to explain how the kinds of conflicts we deal with are basically designed into the organizational structure.

Degrees of Centralism in Governance

Before talking about Gentoo I want to contrast a few forms of democratic governance and the role of centralism in avoiding or fostering stalemate. Democratic governments vary greatly in the number of centers of power with different constituencies.

The most extreme centralization of power lies in states that use a unicameral unitary parliamentary system, such as Norway (and most corporations/organizations). In such a state the people vote for parties to form a parliament, and that parliament in turn exercises nearly complete control over all aspects of the government. They appoint ministers, and any governance bodies in the state derive their power from the parliament. Such organizations tend to have little internal division, as the legislature has the power to overcome almost any stalemate (the courts are usually an exception which I’ll set aside for now). Some definitions:

Unicameral – power is invested in a single legislative body.

Unitary – a single layer of government exists.

Parliamentary – the executive power is appointed by the legislature.

An example of a very decentralized system of government is in a bicameral federal presidential system, such as in the United States. In such a state there are many somewhat independent governance bodies which have differing scopes of power and constituencies. No single legislative body has the power to create laws, the executive power is independent, and even acting together these do not have absolute sovereignty over all aspects of the operation of the state. For example, in the current US government different parties hold sway over each of the legislative bodies and the executive branch, and a recent court case upheld the power of states to choose to not enact some elements of the Affordable Care Act. Individual US voters have differing levels of power in electing officials to each governing body based on geography and state population. Such organizations tend to have quite a bit of internal division. Some definitions:

Bicameral – power is invested in two legislative bodies which usually have to collaborate to pass a law.

Federal – multiple layers of semi-independent government exist.

Presidential – the executive power is directly elected.

Governance Bodies in Gentoo

Now let’s examine the governance bodies that exist within Gentoo.

There are 5 special projects and a large number of general projects. Here are the scope and constituencies of each of the special projects:

The Council is tasked with decisions that affect the technical design/operation of the distribution. It is elected by all developers annually. It is the court of appeals for QA and Comrel.

The Trustees are tasked with the legal operation of the Gentoo Foundation, which owns the IP of Gentoo and manages the money. It is elected by all foundation members biannually.

QA is tasked with the quality of the distribution and has enforcement powers. Its lead is elected annually by its members, and members are appointed by the lead.

Comrel is tasked with dispute resolution and discipline and has enforcement powers. Its lead is elected annually by its members, and members are appointed by the lead.

Infra is tasked with keeping the lights on, both for Gentoo-owned hardware and coordinating/managing resources donated to our use. Its lead is elected annually by its members, and members are appointed by the lead.

Then there are a multitude of normal projects. Anybody can join a normal project, and they all elect their leads annually. In addition to anybody being able to join a normal project, anybody else can also start a competing normal project.

I’ll set aside the normal projects, because for the most part I don’t think they are the cause of many governance issues. Their power is very limited, and they are very open for any developer to join. Instead I’ll focus on the special projects.

If you look at the 5 special projects, every single one of them has a different constituency. Three have a constituency consisting solely of their own members, and the two elected bodies have largely overlapping but not identical constituencies. Since they have different constituencies, it is entirely possible for them to be in long-term conflict. Since no one body has power over the others, there is also the potential for stalemate.

Where We Should Be, and How to Get There

I would advocate that Gentoo would be better served if it had fewer independent governance bodies, and if there were clear hierarchy between them. Less independence means that there is less potential for conflict, and hierarchy means that when there is conflict there is less potential for stalemate.

That said, I still think it makes sense to preserve these organizations – they really do have different roles in sustaining Gentoo and different skill-sets are needed to make them work. I also don’t think that we need to change everything in the span of a day.

Here is my outline of how I think Gentoo should operate, eventually:

The Trustees are the legal board of directors for the distribution. The Foundation owns everything about Gentoo that can be legally owned. Legally the Foundation IS Gentoo. In practice I would not suggest changing the scope of their operations, but legally they would have the power to resolve any matter as long as they are consistent with the Charter/Bylaws.

The Council is tasked with leading the technical operation of the distribution. It would be legally subordinate to the Trustees in the event that the Foundation believes that the Council is causing Gentoo to violate the law or its Charter, but it would be otherwise self-governing and elected by the developers. The Council would be the court of appeal for any enforcement action, and would additionally have the power to step in and resolve disputes among projects. All projects are sub-ordinate to either the Council, or directly to the Trustees.

QA, Devrel, and Infra would continue to appoint their own members and elect a lead, and would maintain the same scope/powers. However, the leads of each would need to be confirmed by the Council, and could be replaced by a decision of the Council. The leads would also be required to report status periodically to the Council.

Individual projects are all subservient to the Council/Trustees in principle, but would be largely hands-off as long as they allow for open membership, do not interfere with each other, and elect their leads annually.

I could see debate over whether Devrel or Infra should report to the Trustees directly vs to the Council. I really don’t have a strong opinion there and you could make a good case for either.

In the short term I see QA as the best place to start as it is an organization that is going to be reconstituted in any case, and the new membership could help set a model for the other special projects to follow. Infra is functioning very well currently and would probably be the last thing I’d advocate touching, with the goal of minimal disruption. Devrel falls somewhere in-between, but in my proposal all special projects still select their leads and it is the role of the Council to accept them – the Council stepping in and interfering with the constitution of a special project should be used only to resolve serious disputes.

So, that was long-winded as usual. I’d be very interested in reactions/thoughts. I’d also be interested in examples of how models like the one I propose have been successful or have failed.

Filed under: foss, gentoo, gentoo foundation, linux]]>https://rich0gentoo.wordpress.com/2013/11/17/the-balance-of-power-in-gentoo/feed/8rich0MythTV 0.26 In Portagehttps://rich0gentoo.wordpress.com/2013/01/23/mythtv-0-26-in-portage/
https://rich0gentoo.wordpress.com/2013/01/23/mythtv-0-26-in-portage/#commentsThu, 24 Jan 2013 01:31:15 +0000http://rich0gentoo.wordpress.com/?p=197]]>Well, all of MythTV 0.26 is now in portage, masked for testing for a few days.

If anyone is interested now is a good time to give it a try and report any issues you find. If all is quiet the masks will come off and we’ll be up-to-date (including all patches up to a few days ago).

Thanks to all who have contributed to the 0.26 bug. I can also happily report that I’m running Gentoo on my mythtv front-end, which should help me with maintaining things. MiniMyth is a great distro, but it has made it difficult to keep the front- and back-ends in sync.

Filed under: foss, gentoo, mythtv]]>https://rich0gentoo.wordpress.com/2013/01/23/mythtv-0-26-in-portage/feed/7rich0Gentoo and Copyright Assignmentshttps://rich0gentoo.wordpress.com/2012/12/15/gentoo-and-copyright-assignments/
https://rich0gentoo.wordpress.com/2012/12/15/gentoo-and-copyright-assignments/#commentsSat, 15 Dec 2012 13:43:38 +0000http://rich0gentoo.wordpress.com/?p=192]]>A topic that has been fairly quiet for years has roared into life on a few separate occasions in the last month within the Gentoo community: copyright assignments. The goal of this post is to talk a little about the issues around these as I see them. I’ll state upfront that I’m not married to any particular approach.

But first, I think it is helpful to consider why this topic is flaring up. The two situations I’m aware of where this has come up in the last month or so both concern contributions (willing or not) from outside of Gentoo. One concerns a desire to be able to borrow eclass code from downstream distros like Exherbo, and the other is the eudev fork. In both cases the issue is with the general Gentoo policy that all Gentoo code have a statement at the top to the effect of “Copyright 2012 Gentoo Foundation.”

Now, Diego has already blogged about some of the issues created by this policy already, and I want to set that aside for the moment. Regardless of whether the Foundation can lay claim to ownership of copyright on past contributions, the question remains, should Gentoo aim to have copyright ownership (or something similar) for all Gentoo work be owned by the Foundation?

Right now I’m reaching out to other free software organizations to understand their own policies in this area. Regardless of whether we want to have Gentoo own our copyrights or not there are still legal questions around what to put on that copyright line, especially when a file is an amalgamation of code originated both inside and outside of Gentoo, perhaps even by parties who are hostile to the effort. I can’t speak for the Trustees as a whole, but I suspect that after gathering info we’ll try to have some open discussion on the lists, and perhaps even have a community-wide vote before making new policy. I don’t want to promise that – in fact I’d recommend that any community-wide vote be advisory only unless a requirement for supermajority were set, as I don’t want half the community up in arms because a 50.1% majority passed some highly unpopular policy.

So, what are some of the directions in which Gentoo might go? Why might we choose to go in these directions? Below I outline some of the options I’m aware of:

Maintain the status quo
We could just leave the issue of copyright assignment somewhat ambiguous as has been done. If Gentoo were forced to litigate over copyright ownership right now an argument could be made that because contributors willingly allowed us to stick that copyright notice on our files and made their contribution with the knowledge of our policies, that they have given implicit consent to our doing so.

I’m not a big fan of this approach – it has the virtue of requiring less work, but really has no benefits one way or the other (and as you’ll read below their are benefits from declaring a position one way or the other).

This requires us to come up with a policy around what goes on the copyright notice line. I suspect that there won’t be much controversy for Gentoo-originated work like most ebuilds, as there isn’t much controversy over them now. However, for stuff like eudev or code borrowed from other projects this could get quite messy. With no one organization owning much of the code in any file the copyright line could become quite a mess.

Do not require copyright assignment
We could just make it a policy that Gentoo would aim to own the name Gentoo, but not the actual code we distribute. This would mean that we could freely accept any code we wished (assuming it was GPL or CC BY-SA compatible per our social contract). This would also mean that Gentoo as an organization would find it difficult to pursue license violations, and future relicensing would be rather difficult.

From an ability to merge outside code this is clearly the preferred solution. This approach still carries all the difficulties of managing the copyright notice, since again no one organization is likely to hold the majority of copyright ownership of our files. Also, if we were to go this route we should strongly consider requiring that all contributions be licensed under GPL v2+, and not just GPL v2. Since Gentoo would not own the copyright if we ever wanted to move to a newer GPL version we would not have the option to do so unless this were done.

Gentoo would still own the name Gentoo, so from a branding/community standpoint we’d have a clear identity. If somebody else copied our code wholesale the Foundation couldn’t do much to prevent this unless we retroactively asked a bunch of devs to sign agreements allowing us to do so, but we could keep an outside group from using the name Gentoo, or any of our other trademarks.

Require copyright assignment
We could make it a policy that all contributions to Gentoo be made in conjunction with some form of copyright assignment, or contributor licensing agreement. I’ll set aside for now the question of how exactly this would be implemented.

In this model Gentoo would have full legal standing to pursue license violations, and to re-license our code. In practice I’m not sure how likely we’d actually be to do either. The copyright notice line would be easy to manage, even if we made the occasional exception to the policy, since the exceptions could of course be managed as exceptions as well. Most likely the majority of the code in any file would only be owned by a few entities at most.

The downside to this approach is that it basically requires turning away code, or making exceptions. Want to fork udev? Good luck getting them to assign copyright to Gentoo.

There could probably be blanket exceptions for small contributions which aren’t likely to create questions of copyright ownership. And we could of course have a transition policy where we accept outside code but all modifications must be Gentoo-owned. Again, I don’t see that as a good fit for something like eudev if the goal is to keep it aligned with upstream.

I think the end result of this would be that work that is outside of Gentoo would tend to stay outside of Gentoo. The eudev project could do its thing, but not as a Gentoo project. This isn’t necessarily a horrible thing – OpenRC wasn’t really a “Gentoo project” for much of its life (I’m not quite sure where it stands at the moment).

Alternatives
There are in-between options as well, such as encouraging the voluntary assignment/licensing of copyright (which is what KDE does), or dividing Gentoo up into projects we aim to own or not. So, we might aim to own our ebuilds and the essential eclasses and portage, but maybe there is the odd eclass or side project like eudev that we don’t care about owning. Maybe we aim to own new contributions (either all or most).

There are good things to be said for a KDE-like approach. It gives us some of the benefits of attribution, and all of the benefits of not requiring attribution. We could probably pursue license violations vigorously as we’d likely hold control of copyright over the majority of our work (aside from things like eudev – which obviously aren’t our work to begin with). Relicensing would be a bit of a pain – for anything we have control over we could of course relicense it, but for anything else we’d have to at least make some kind of effort to get approval. Legally that all becomes a murky area. If we were to go with this route again I’d probably suggest that we require all code to be licensed GPL v2+ or similar just to give us a little bit of automatic flexibility.

I’m certainly interested in feedback from the Gentoo community around these options, things I hadn’t thought of, etc. Feel free to comment here or on gentoo-nfp.

Filed under: foss, gentoo, gentoo foundation]]>https://rich0gentoo.wordpress.com/2012/12/15/gentoo-and-copyright-assignments/feed/10rich0The Dark Side of Qualityhttps://rich0gentoo.wordpress.com/2012/12/06/the-dark-side-of-quality/
https://rich0gentoo.wordpress.com/2012/12/06/the-dark-side-of-quality/#commentsThu, 06 Dec 2012 15:48:32 +0000http://rich0gentoo.wordpress.com/?p=182]]>Voltaire once said that the best is the enemy of the good. I think that there are few places where one can see as many abuses of quality as you’ll find in many FOSS projects, including Gentoo.

Often FOSS errs on the side of insufficient quality. Developers who are scratching itches don’t always have incentive to polish their work, and as a result many FOSS projects result in a sub-optimal user experience. In these cases “good enough” is standing in the way of “the best.”

However, I’d like to briefly comment on an opposite situation, where “the best” stands in the way of “good enough.” As an illustrative example, consider the excellent practice of removing bundled libraries from upstream projects. I won’t go on about why this is a good thing – others have already done so more extensively. And make no mistake – I agree that this is a good thing, the following notwithstanding.

The problem comes when things like bundled libraries become a reason to not package software at all. Two examples I’m aware of where this has happened recently are media-sound/logitechmediaserver-bin and media-gfx/darktable. In the former there is a push to remove the package due to the inclusion of bundled libraries. In the latter the current version is lagging somewhat because while upstream actually created an ebuild, it bundles libraries. Another example is www-client/chromium, which still bundles libraries despite a very impressive campaign by the chromium team to remove them.

The usual argument for banning packages containing bundled libraries is that they can contain security problems. However, I think this is misleading at best. If upstream bundles zlib in their package we cry about potential security bugs (and rightly so), however, if upstream simply writes their own compression functions and includes them in the code, we don’t bat an eyelash, even though this is more likely to cause security problems. The only reason we can complain about zlib is BECAUSE it is extensively audited, making it easy to spot the security problems. We’re not reacting to the severity of problems, but only to the detectablity of them.

Security is a very important aspect of quality, but any reasonable treatment of security has to consider the threat model. While software that bundles a library is rightfully considered “lower” in quality than one that does not, what matters more is whether this is a quality difference that is meaningful to end users, and what their alternatives are. If the alternative for the user is to just install the same software with the same issues, but from an even lower quality source with no commitment to security updates, then removing a package from Gentoo actually increases the risks to our users. This is not unlike the situation that exists with SSL, where an unencrypted connection is presented to the user as being more secure than an SSL connection with a self-signed certificate, when this is not true at all. If somebody uses darktable to process photos that they take, then they’re probably not concerned with a potential buffer overflow in a bundled version of dcraw. If the another user operated a service that accepted files from strangers on the internet, then they might be more concerned.

What is the solution?: A policy that gives users reasonably secure software from a reputable source, with clear disclosure. We should encourage devs to unbundle libraries, consider bugs pointing out bundled libraries valid, accept patches to unbundle libraries when they are available, and add an elog notice to packages containing bundled libraries in the interest of disclosure. Packages with known security vulnerabilities would be subject to the existing security policy. However, developers would still be free to place packages in the tree that contain bundled libraries, unmasked, and they could be stabilized. Good enough for upstream should be good enough for Gentoo (again, baring specific known vulnerabilities), but that won’t stop us from improving further.

Second, I’d like to introduce a few enhancements I’ve made on these (some being merged upstream already).

Third, I’d like to turn this into a bit of a tutorial into getting started with EC2 as well since these scripts make it brain-dead simple.

I’ve previously written on building a Gentoo EC2 image from scratch, but those instructions do not work on EBS instances without adjustment, and they’re fairly manual. Edowd extended this work by porting to EBS and writing scripts to build a gentoo install from a stage3 on EC2. I’ve further extended this by adding a rudimentary plugin framework so that this can be used to bootstrap servers for various purposes – I’ve been inspired by some of the things I’ve seen done with Chef and while that tool doesn’t fit perfectly with the Gentoo design this is a step in that direction.

What follows is a step-by-step howto that assumes you’re reading this on Gentoo and little else, and ends up with you at a shell on your own server on EC2. Those familiar with EC2 can safely skim over the early parts until you get to the git clone step.

To get started, go to aws.amazon.com, and go through the steps of creating an account if you don’t already have one. You’ll need to specify payment details/etc. If you buy stuff from amazon just use your existing account (if you want), and there isn’t much more than enabling AWS.

Log into aws.amazon.com, and from the top right corner drop-down under either your name or My Account/Console choose “Security Credentials”.

Browse down to access credentials, click on the X.509 certificate tab, generate a certificate, and then download both the certificate and private key files. The web services require these to do just about anything on AWS.

On your gentoo system run as root emerge ec2-ami-tools ec2-api-tools. This installs the tools needed to script actions on EC2.

Export into your environment (likely via .bashrc) EC2_CERT and EC2_PRIVATE_KEY. These should contain the paths to the files you created in the previous step. Congratulations – any of the ac2-api-tools should now work.

We’re now going to checkout the scripts to build your server. Go to an empty directory and run git clone git://github.com/rich0/rich0-gentoo-bootstrap.git -b rich0-changes.

chdir to the repository directory if necessary, and within it run ./setup_build_gentoo.sh. This creates security zones and ssh keys automatically for you, and at the end outputs command lines that will build a 32 or 64 bit server. The default security zone will accept inbound connections to anywhere, but unless you’re worried about an ssh zero-day that really isn’t a big deal.

Run either command line that was generated by the setup script. The parameters tell the script what region to build the server in, what security zone to use, what ssh public key to use, and where to find the private key file for that public key (it created it for you in the current directory).

Go grab a cup of coffee – here is what is happening:

A spot request is created for a half decent server to be used to build your gentoo image. This is done to save money – amazon can kill your bootstrap server if they need it, and you’ll get the prevailing spot rate. You can tweak the price you’re willing to pay in the script – lower prices mean more waiting. Right now I set it pretty high for testing purposes.

The script waits for an instance to be created and boot. The build server right now uses an amazon image – not Gentoo-based. That could be easily tweaked – you don’t need anything in particular to bootstrap gentoo as long as it can extract a stage3 tarball.

A few build scripts are scp’ed to the server and run. The server formats an EBS partition for gentoo and mounts it.

A stage3 and portage snapshot are downloaded and extracted. Portage config files (world, make.conf, etc) are populated. A script is created inside the EBS volume, and executed via chroot.

That script basically does the typical handbook install (emerge sync, update world (which has all the essentials in it like dhcpcd and so on), build a kernel, configure rc files, etc.

The bootstrap server terminates, leaving behind the EBS volume containing the new gentoo image. A snapshot is created of this image and registered as an AMI.

A micro instance of the AMI is launched to test it. After successful testing it is terminated.

After the script is finished check the output to see that the server worked. If you want it outputs a command line to make the server public – otherwise only you can see/run it.

To run your server go to aws.amazon.com, sign in if necessary, browse to the EC2 dashboard. Click on AMIs on the left side, select your new gentoo AMI, and launch it (micro instances are cheap for testing purposes). Go to instances on the left side and hit refresh until your instance is running. Click on it and look down in the details for the public DNS entry.

To connect to your instance run ssh -i <path to pem file in your bootstrap directory> ec2-user@<public DNS name of your server>. You can sudo to root (no password).

That’s it – you have a server in the cloud. When you’re done be sure to clean up to avoid excessive charges (a few cents an hour can add up). Check the instances section and TERMINATE (not stop) any instances that are there. You will be billed by the month for storage so de-register AMIs you don’t need and go to the snapshot section and delete their corresponding snapshots.

Now, all that is useful, but you probably want to tailor your instance. You can of course do that interactively, but if you want to script it check out the plugins in the plugin directory. Just add a path to a plugin file at the end of the command line to build the instance and it will tailor your image accordingly. I plan to clean up the scripts a bit more to move anything discretionary into the plugins (you don’t NEED fcron or atop on a server).

The plugins/desktop plugin is a work in progress, but I think it should work now (takes the better part of a day to build). It only works 32-bit right now due to the profile line. However, if you run it you should be able to connect with x2goclient and have a KDE virtual desktop. A word of warning – a micro instance is a bit underpowered for this.

And on a side note, if somebody could close bugs 427722 and 423855 that would eliminate two hacks in my plugin. The stable NX doesn’t work with x2go (I don’t know if it works for anything else), and the stable gst-plugins-xvideo is missing a dependency. The latter bug will bite anybody who tries to install a clean stage3 and emerge kde-meta.

All of this is very much a work in progress. Patches or pull requests are welcome, and edowd is maintaining a nice set of up-to-date gentoo images for public use based on his scripts.

Note: Recently I modified this script to add my dev overlay. I’m mainly doing this to deal with build failures. However, in the interest of full disclosure I want to make sure all are aware.