Commit Message

From: Kim Højgaard-Hansen <kiho@prevas.dk>
The hostprep command can show what command is needed to prepare the
host for OE-lite. A file for each supported host configuration must
be added to th supported-host-configs directory. The file should
contain the command needed to install the dependencies needed for
using oe-lite. The file name should be: distro_release_codename
as defined by lsb_release.
The lsb_release tool is used for automatic host detection. A
list of the supported host configurations can be shown by passing
-l to the hostprep command. If only the preparation command for the
current host should be shown, run 'hostprep -c'
---
lib/oelite/cmd/__init__.py | 2 +-
lib/oelite/cmd/hostprep.py | 132 +++++++++++++++++++++
.../cmd/supported-host-configs/centos_6.4_final | 13 ++
.../cmd/supported-host-configs/exherbo_none_none | 1 +
.../supported-host-configs/ubuntu_12.04_precise | 7 ++
5 files changed, 154 insertions(+), 1 deletion(-)
create mode 100644 lib/oelite/cmd/hostprep.py
create mode 100644 lib/oelite/cmd/supported-host-configs/centos_6.4_final
create mode 100644 lib/oelite/cmd/supported-host-configs/exherbo_none_none
create mode 100644 lib/oelite/cmd/supported-host-configs/ubuntu_12.04_precise

Kim Højgaard-Hansen <kiho@prevas.dk> writes:
> On 12/09/13 12:13, Esben Haabendal wrote:>> <kim.hansen@prevas.dk> writes:>>> From: Kim Højgaard-Hansen <kiho@prevas.dk>>> The hostprep command can show what command is needed to prepare the> host for OE-lite. A file for each supported host configuration must> be added to th supported-host-configs directory. The file should> contain the command needed to install the dependencies needed for> using oe-lite. The file name should be: distro_release_codename> as defined by lsb_release.> The lsb_release tool is used for automatic host detection. A> list of the supported host configurations can be shown by passing> -l to the hostprep command. If only the preparation command for the> current host should be shown, run 'hostprep -c'>> I think we should rename the command to 'setup'.>> no problem, setup is just quite overloaded in my opinion>>> Also, for ease of use, I think that the most likely usage should be as> obvious as possible.>> The typical use of this feature should be to actually run the commands> needed to setup the host for OE-lite development, and should be> supported by something as simple as>> oe setup>> I did not think we would let oe actually run the command. If we want> that we need to be really sure we detected the correct host> configuration. I (also as a user) would prefer to run any commands> myself, but just be told what to do, then I can inspect it.
Good point. But at the same time, we want to make it as convenient and
straight-forward to do the most simple and usual tasks.
Perhaps making "oe setup" print out to console what it is to be
executed, and ask the user to confim with 'yes/no' (default to no, and
require to type 'yes' and not just 'y'), and then add a '-y' option to
skip that and just execute the command(s).
When running "oe setup" without '-y', you could even check if it is
actually a tty, and if not, just revert to do the same as if '-p' was
given.
> My idea of printing out the detected host configuration would be to avoid> problems of being "too clever", guess that could also be done by prompting for> a user reply like when we run "oe bake" ?
Yes, and there we also provide the '-y' :)
But, it differs in a couple of ways.
It is quite common end in situations where "oe bake" wants to build more
than you wanted, and thus you often end up answering 'no', and then
doing some kind of '--sloppy' bake instead. For "oe setup", _if_ we
provide a reliable host distro detection system, most (if not all) users
will want to answer 'yes'.
Running "oe bake" is something you do all the time when working with
OE-lite, while "oe setup" should be very rare.
Running "oe setup" changes your host OS, where "oe bake" only changes
your OE-lite tmp dir.
The last two differences points into the direction of requiring user
confirmation, where the first points in the opposite direction.
So overall, it is probably best to not just execute the command(s), but
an '-y'/'--yes' option should be supported.
> Listing supported host configuration/distros could be kept as>> oe setup -l>> And for printing out the commands could be kept as:>> oe setup -p>> but changed to print out in a format that can be piped directly into a> shell, so something like this can be done:>> oe setup -p | /bin/sh>> Non-executable information like host configuration can be printed as '#'> comment lines.>> sure, makes sense>>>> ---> lib/oelite/cmd/__init__.py | 2 +-> lib/oelite/cmd/hostprep.py | 132 +++++++++++++++++++++> .../cmd/supported-host-configs/centos_6.4_final | 13 ++> .../cmd/supported-host-configs/exherbo_none_none | 1 +> .../supported-host-configs/ubuntu_12.04_precise | 7 ++> 5 files changed, 154 insertions(+), 1 deletion(-)> create mode 100644 lib/oelite/cmd/hostprep.py> create mode 100644 lib/oelite/cmd/supported-host-configs/centos_6.4_final> create mode 100644 lib/oelite/cmd/supported-host-configs/exherbo_none_none> create mode 100644 lib/oelite/cmd/supported-host-configs/ubuntu_12.04_precise>> diff --git a/lib/oelite/cmd/__init__.py b/lib/oelite/cmd/__init__.py> index d2feabe..6919262 100644> --- a/lib/oelite/cmd/__init__.py> +++ b/lib/oelite/cmd/__init__.py> @@ -1 +1 @@> -manifest_cmds = [ "bake", "show", "cherry" ]> +manifest_cmds = [ "bake", "hostprep","show", "cherry" ]> diff --git a/lib/oelite/cmd/hostprep.py b/lib/oelite/cmd/hostprep.py> new file mode 100644> index 0000000..ae82fd8> --- /dev/null> +++ b/lib/oelite/cmd/hostprep.py> @@ -0,0 +1,132 @@> +import oebakery> +from oelite import util> +import logging> +import os> +import subprocess> +import oelite> +> +description = "Host preparation tool"> +arguments = ()> +> +def add_parser_options(parser):> + parser.add_option("-c", "--command",> + action="store_true", default=False,> + help="Show the command needed to prepare the> host for OE-lite")>> Change to '-p', '--print'>> ok>>>> + parser.add_option("-l", "--list-supported-host-configs",>> Maybe choose a slightly shorter long options, fx. just '--list'.>> ok>>>> + action="store_true", default=False,> + help="Show a list of known host configurations")> + parser.add_option("-s", "--select-host-config",>> Again, a shorter long option like '--host' is preferable. And in that> case, '-h' would probably be a better short option.>> ok>> + action="store_true", default=False,> + help="Manually select a host configuration")> + parser.add_option("-d", "--debug",> + action="store_true", default=False,> + help="Debug the OE-lite metadata")>> The help text is a bit misleading, as the metadata is not actually used> in this command, and it is only the command itself that is debugged.>> yeah that was copied directly from another command :)>> + return> +> +> +def parse_args(options, args):> + if options.debug:> + logging.getLogger().setLevel(logging.DEBUG)> + else:> + logging.getLogger().setLevel(logging.INFO)> + if args:> + options.head = args.pop(0)> + else:> + options.head = None> + return> +> +def get_supported_host_configs():> + logging.debug("Reading supported host configurations")> + #TODO: fix path handling, find place to put host configs>> This needs to be implemented. You should be able to use> bb.utils.which() to find files and/or dirs in all layers.>> To search for a specific host config file, use something like this:>> bb.utils.which(d.get('OEPATH'), 'setup/debian_7.1')>> or to search for all 'setup/' dirs:>> bb.utils.which(d.get('OEPATH'), 'setup')>> perfect>> + hid = 0> + hc=[]> + config_dir = "./meta/core/lib/oelite/cmd/supported-host-configs"> + logging.debug("Using config directory: %s", config_dir)> + for config in os.listdir(config_dir):> + logging.debug("Reading config: %s", config)> + keys = config.split('_')> + if(len(keys) != 3):> + logging.debug("Ignoring host-config with wrong name: %s", config)> + continue> + #read out the cmd to run for this host configuration> + try:> + logging.debug("Read the host preparation command")> + #strip off the last newline> + cmd = open(config_dir+"/"+config, 'r').read()[:-1]> + except:> + logging.debug("Reading the host preparation command from the host configuration file failed")> + raise> + hc.append(dict(distro=keys[0], release=keys[1],> codename=keys[2], hostID=hid, command=cmd))>> Any special reason to use both distro, release and codename? Won't> distro and release be enough? Or do you know of distributions that use> codenames and not release versions?>> to be sure it does not cause any problems mainly, none of these are actually> mandated so it is a bit fragile no matter how you look at it. if it matters to> you we can just drop the codename
I would prefer to drop the codename for now.
It makes it less obscure to add/maintain host configurations.
> + hid+=1> + logging.debug("Adding host config for unknown host")> + hc.append(dict(distro="unknown", release="none", codename="none", hostID=hid, command="Unknown host - Please refer to the installation documentation"))> + return hc>> Generally, I don't see why it would be necessary to get a list of all> supported host configs unless we are listing them. Instead, when> auto-detecting or when looking for a specified host configuration, you> should just search for the setup file using bb.util.which().>> does it matter if that list is built up first, then searched ? But sure we can> change that
It might not matter, but I think the code will be smaller and more clear
if you do it that way. It is quite simple to use the bb.utils.which()
command to search for the host configuration, and you don't have to care
about inventing hostID or something like that and implementing a search
loop (not that it is complicated, but less code is often better).
> And while we are at the searching topic, we probably need to have some> major/minor version search logic.>> In my case, I have Debian 7.1 installed. We should probably search for> 'debian_7.1' first, and if not found, try 'debian_7' next, and then> 'debian', and finally 'unknown'.>> uh that last bit sounds a bit dangerous to me, if you just want it to run the> command it finds.
It might be, but in this case, you really don't want to do things
different for 7.1 than for 7.0. And, if we do this in the strict way,
an old OE-lite manifest is almost guaranteed to not work out of the box
in the future, as releases such as Debian will have bumbed their minor
versions. And this is opposite of what we try to achieve.
Of-course, you probably do not want to have a 'debian' host
configuration available, but there might be other cases where a version
independent host configuration might make sense (rolling-release like
exherbo springs to mind...)
> +> +def determine_host(host_configs):> + logging.debug("Running host detection")> + logging.debug("Searching for lsb_release information")> + #try lsb-release which is the most widely available method> + if(os.path.exists("/etc/lsb-release") and os.path.isfile("/etc/lsb-release")):>> Why look for the /etc/lsb-release file? I don't think that LSB> specifies this file, but the lsb_release command is mandatory.>> Just look for the lsb_release binary.>> "If the installation is LSB compliant, the "/etc/lsb-release" file should> contain the LSB_VERSION field",
Yes, but for distros, such as Debian, that are not LSB compliant and
does not comply to any LSB modules does not have this file.
And in this case, we don't use this information for anything, we just
need the information that the lsb_release command can provide.
> however you might be right that it would be quite weird to have the> lsb_release binary without that information, and if so it would just> report nothing.
My Debian 7.1 install does not have the /etc/lsb-release file, but does
have the lsb_release command. Because of the missing /etc/lsb-release I
get:
~ $ lsb_release -v
No LSB modules are available.
Which is correct, and for now, we really don't care about this.
> + try:> + logging.debug("checking for the lsb_release binary")> + subprocess.call(["lsb_release"])> + except OSError as e:> + if e.errno == os.errno.ENOENT:> + # handle file not found error.> + logging.debug("/etc/lsb-release found, but lsb_release binary not available")> + return (host for host in host_configs if host["distro"] == "unknown").next()> + else:> + logging.debug("unhandled exception when calling lsb_release")> + raise> + else:> + try:> + logging.debug("using lsb_release to get host information")> + #strip newlines and make lowercase for later matching> + distro = oelite.util.shcmd("lsb_release -si", quiet=True)[:-1].lower()> + release = oelite.util.shcmd("lsb_release -sr", quiet=True)[:-1].lower()> + codename = oelite.util.shcmd("lsb_release -sc",> quiet=True)[:-1].lower()>> As mentioned above, I think we can drop the codename part.>> ok>>>> + except:> + logging.debug("unhandled exception when calling lsb_release")> + raise> + else:> + logging.debug("matching lsb_release information to supported configs")> + logging.debug("lsb_release: distro: %s | release: %s | codename: %s", distro, release, codename)> + for host in host_configs:> + if((distro == host["distro"]) and (release == host["release"]) and (codename == host["codename"])):> + logging.debug("found matching host config with ID: %d", host["hostID"])> + return host> + logging.debug("Could not match information from lsb_release to any known host config")> + logging.debug("Checking for Exherbo")> + if(os.path.exists("/etc/exherbo-release") and os.path.isfile("/etc/exherbo-release")):> + return (host for host in host_configs if host["distro"] == "exherbo").next()> + logging.debug("No host config found for this host")> + return (host for host in host_configs if host["distro"] == "unknown").next()> +> +def print_host_config(host):> + print> '{0:8}{1:15}{2:9}{3:15}{4:10}{5:15}{6:22}{7:2}'.format("Distro:",> host["distro"], "Release:", host["release"], "Codename:",> host["codename"], "Host Configuration ID:", host["hostID"])>> I don't see the point in printing out hostID, it is just an> implementation detail of this command.>> this was for the case where we can't find a matching config, then the user> might want to print the config that matches "most", so the print command could> optionally take a host ID parameter
Ok. But most people will find it a lot easier to remember something like
ubuntu_12.04 than 372 or some other random number assigned to the closes
match.
> Also, the output claims to print out "List of supported host> configurations", and includes "unknown"... Perhaps just change the> description to "List of known host configurations" or something better.>> yeah, it should not output unknown there, will change the text>>> Also, it might be convenient to have a way of storing information about> known unsupported host distros.>> So if we know that Ubuntu 8.04 will not work, or at least that we know> that our guidelines are not sufficient to make it work, we should have a> host configuration/setup file encode this information.>> Perhaps something like this in setup/ubuntu_8.04:>> # UNSUPPORTED> echo "This OE-lite manifest is not supported on Ubuntu 8.04"> exit 1>>> ok, we can add that, but it could easily be added later if needed no?
Yes, that can be added later.
> do we have a known case of this already?
Yes, I have had users trying to use OE-lite on outdated Ubuntu and
Fedora hosts.
> would be a large list if we have to list all older then FOO :)
Hehe, of-course. And we should not try to add files for all old distro
versions in the world. But it would be nice to have best possible error
messages to developers using fx. Ubuntu 8.04.
> +def run(options, args, config):> + logging.debug("hostprep.run %s", options)> +> + hc = get_supported_host_configs()>> As mentioned above, you don't actually need to get this information> unless the '-l' argument is given.>> agreed>>>> +> + if(options.list_supported_host_configs):> + print "List of supported host configurations:\n"> + for host in hc:> + print_host_config(host)> + return 0> +> + #try automatic host detection> + host = determine_host(hc)>> Don't you need to add support for manual host selection here?>> yes, did not finish that, would do that for the print command though>>>> + if(options.command):> + logging.debug("printing command for hostID: %d", host["hostID"])> + logging.debug("Command: '%s'", host["command"])> + print host["command"]> + return 0> + print "Your host configuration detected to be:\n"> + print_host_config(host)> + print> + print "Run this command on your host to prepare it for OE-lite:\n"> + print host["command"]> +> + return 0> diff --git a/lib/oelite/cmd/supported-host-configs/centos_6.4_final b/lib/oelite/cmd/supported-host-configs/centos_6.4_final> new file mode 100644> index 0000000..2da269f> --- /dev/null> +++ b/lib/oelite/cmd/supported-host-configs/centos_6.4_final> @@ -0,0 +1,13 @@> +wget http://mirrors.nl.eu.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm \>> Would it be possible to use a non-mirror URL instead? It would be nice> to not be affected by changes to mirrorring details for the EPEL project.>> basically that first part should not be there. It does not make sense to add> stuff that is needed for installing "oe" in this setup command :)
Ok. Which EPEL parts do we need for installing OE-lite Bakery?
> +&& sudo yum install epel-release-6-8.noarch.rpm \> +&& sudo yum install git python-ply python-setuptools \> +&& wget -qO- http://oe-lite.org/download/bakery/oe-lite-bakery-4.0.2.tar.gz \> + | tar -xz && cd oe-lite-bakery-4.0.2 && sudo python setup.py install \> +&& sudo yum groupinstall "Development Tools" \> +&& sudo yum install python-sqlite2 python-magic python-pycurl python-devel \> +fakeroot gettext-devel ncurses-devel libtool texinfo flex bison \> +coreutils sed git-core cvs subversion mercurial quilt gawk texinfo \> +automake autoconf curl openjade groff make gcc-c++ gcc binutils bc \> +unzip gtk-doc docbook-utils xmlto glib2-devel intltool glibc-static \> +gperf>> Why not add>> set -e>> As first command, and avoid all the '&&' cruft?>> Actually, maybe we should just generally prepend 'set -ex' to all the> host configuration files before running them (excluding those marked as> 'UNSUPPORTED' to avoid echo command littering).>> sure>>>> +> diff --git a/lib/oelite/cmd/supported-host-configs/exherbo_none_none b/lib/oelite/cmd/supported-host-configs/exherbo_none_none> new file mode 100644> index 0000000..2161776> --- /dev/null> +++ b/lib/oelite/cmd/supported-host-configs/exherbo_none_none> @@ -0,0 +1 @@> +sudo cave resolve oe-bakery> diff --git a/lib/oelite/cmd/supported-host-configs/ubuntu_12.04_precise b/lib/oelite/cmd/supported-host-configs/ubuntu_12.04_precise> new file mode 100644> index 0000000..98c533c> --- /dev/null> +++ b/lib/oelite/cmd/supported-host-configs/ubuntu_12.04_precise> @@ -0,0 +1,7 @@> +sudo apt-get install python python-support python-magic python-ply \> +python-pycurl python-pysqlite2 python-pkg-resources python-dev \> +coreutils sed git-core cvs subversion mercurial quilt gawk texinfo \> +automake autoconf autopoint libtool curl texi2html diffstat \> +openjade groff mtd-utils build-essential make gcc g++ binutils \> +bison flex bc ncurses-dev unzip lzma gtk-doc-tools docbook-utils \> +libxml2-utils xmlto help2man libglib2.0-dev lzop gperf python-svn>>
/Esben