Bug#497110: boot loader installation failed when dmraid=true - Debian

This is a discussion on Bug#497110: boot loader installation failed when dmraid=true - Debian ; On Sunday 07 September 2008, Giuseppe Iuculano wrote:
> Giuseppe Iuculano ha scritto:
> > Approximately "dmraid -rD" generates three files, from this we can
> > extrapolate metadata, and with a hex editor we can add the fake
> ...

Re: Bug#497110: improved dmraid support in D-I

On Sunday 07 September 2008, Giuseppe Iuculano wrote:
> Giuseppe Iuculano ha scritto:
> > Approximately "dmraid -rD" generates three files, from this we can
> > extrapolate metadata, and with a hex editor we can add the fake
> > signature to the qemu/virtualbox drive

Let's take this out of the BTS as it is unrelated to the BR.
> Ok,this procedure seems to work for me:
> dmraid -rD and i got:
[...]

*.size is the disk size in sectors.
hda_{0,1,2,3}_sil.dat are all identical, but there are some differences
between .dat files for hda and hdb.
*.offset files are numbers close to the end of the disk, so apparently the
SATA RAID signature is saved 4 times near the end of the disk for
redundancy.

The offsets are probably some fixed distance from the end of the disk.

Question is if the disk size is encoded in the .dat files. IMO the main
reason to use the procedure you devised would be to generate disk images
of a different size (80GB was a bit of a pain for the conversion from
vbox to VirtualBox format for example).

Re: Bug#497110: improved dmraid support in D-I

Frans Pop ha scritto:
> *.size is the disk size in sectors.
> hda_{0,1,2,3}_sil.dat are all identical, but there are some differences
> between .dat files for hda and hdb.
> *.offset files are numbers close to the end of the disk, so apparently the
> SATA RAID signature is saved 4 times near the end of the disk for
> redundancy.

In this case yes (Silicon Image fakeraid), but this isn't true always.

> Question is if the disk size is encoded in the .dat files. IMO the main

Yes, you can find Silicon Image metadata layout on dmraid source,
1.0.0.rc14/lib/format/ataraid/sil.h
> reason to use the procedure you devised would be to generate disk images
> of a different size (80GB was a bit of a pain for the conversion from
> vbox to VirtualBox format for example).

Uhmm, there are some unknown metadata sectors, I think this isn't possible.

Re: Bug#497110: improved dmraid support in D-I

Giuseppe Iuculano ha scritto:
>> reason to use the procedure you devised would be to generate disk images
>> of a different size (80GB was a bit of a pain for the conversion from
>> vbox to VirtualBox format for example).
>
> Uhmm, there are some unknown metadata sectors, I think this isn't possible.

Bug#497110: improved dmraid support in D-I

Greetings all,
As previously noted in this bug, I was the person who implemented the bulk of this work in Ubuntu. Granted, as already discussed, there are bits that need work to be cleared up, but I was likely to get to those in the near future for Ubuntu's bug fixing work for intrepid, due in October.

I apologise for not immediately sending my changes back to Debian. Feature freeze for Ubuntu was on August 28, and I was working up till that day to get everything at a point where it worked. Unfortunately that Thursday and Friday were spent on other important Ubuntu pieces, and I have been away from the computer this last week, so I haven't had a chance to follow up with Debian.

Having said that, I am very happy to see that this is being considered for Lenny, and am willing to work with you all to make sure the code being added is understood, reviewed, and uploaded, as well as making any changes I make for Ubuntu available for Debian now and in the future.

Feel free to ask me any questions you may have about any code I may have written, particularly related to parted itself, as that patch took a while to work out, due to dmraid's design. I will be responding in turn to other messages in this bug relating to dmraid metadata and its collection, as well as other caviats I've noticed in my testing.

I am also happy to test on real hardware, as I have a few PCI cards, as well as a recent intel ICH9 board that I can test such configurations as RAID 5, and possibly RAID10/01 if desired.

Re: Bug#497110: improved dmraid support in D-I

Frans Pop ha scritto:
> The physical disks show as 4GB, but the (correctly detected) dmraid device
> is only 100.8MB in size, which is a bit small for actual use :-P
>
> I get the same if I use the raw images with qemu.
>
> Any ideas?

You are right, also for hpt45x disk size is encoded in the .dat file.
At this point the dat file needs to be modified
(1.0.0.rc14/lib/format/ataraid/hpt45x.h in dmraid source helps on this) with an
hex editor, I don't know if I am able to make this, but I will try.

Re: Bug#497110: improved dmraid support in D-I

On Monday 08 September 2008, Giuseppe Iuculano wrote:
> Frans Pop ha scritto:
> > The physical disks show as 4GB, but the (correctly detected) dmraid
> > device is only 100.8MB in size, which is a bit small for actual use.
>
> You are right, also for hpt45x disk size is encoded in the .dat file.
> At this point the dat file needs to be modified
> (1.0.0.rc14/lib/format/ataraid/hpt45x.h in dmraid source helps on this)
> with an hex editor, I don't know if I am able to make this, but I will
> try.

Got it. After changing 3 bytes starting at offset 12 to '000080', which is
corresponds to 8388608 sectors, everything works perfectly.
4GB is much more convenient for testing in an emulator. Thanks!

Bug#497110: improved dmraid support in D-I

Luke Yelavich writes:
> Having said that, I am very happy to see that this is being
> considered for Lenny, and am willing to work with you all to make
> sure the code being added is understood, reviewed, and uploaded, as
> well as making any changes I make for Ubuntu available for Debian
> now and in the future.

It is nice to have your support on that. Please take a look in
Giuseppe patches and see if you can spot any missing stuff.

Bug#497110: improved dmraid support in D-I

Hello Luke,

Thank you for your mail.

I implemented the current support in the installer during DebConf last
year, more out of annoyance with the continuing stream of requests to
support dmraid than out of any personal interest. By chance someone could
make some hardware available to me for the duration of the conference.
Yes, it was a huge hack (and I've always announced and documented it as
such), but at least it made installing to dmraid an option and it
provided a basis to improve support.

Basically I am quite happy to see dmraid supported properly now, at least
for partitioning. That still leaves bootloader support and bootloader
installation essentially a hack, or do you have improvements for that
too?

I was a bit miffed at first when Giuseppe came up with your patches, but
that lessened somewhat when I saw they were recent. However, given that
Debian/I had done the initial support a mail to our list even just to
announce that you were working on this would have been very much
appreciated. I could possibly even have helped or simplified things for
you in a few cases as obviously I know the partman side quite well.

We're currently still waiting for Otavio to discuss the changes with the
release team and give the go-ahead. After that getting the changes into
SVN and uploaded for the installer should be very quick.

On Monday 08 September 2008, Luke Yelavich wrote:
> as well as making any changes I make for Ubuntu available for Debian now
> and in the future.

Please consider - at least when it comes to changes in the installer
itself - to work directly in the Debian SVN repository. After all,
keeping the diff between Debian/D-I and Ubuntu/D-I small only reduces
effort when Ubuntu rebases on unstable for a new release.

I'm quite sad to see the diff growing now that Colin Watson is less active
when it comes to D-I.
> I am also happy to test on real hardware, as I have a few PCI cards, as
> well as a recent intel ICH9 board that I can test such configurations
> as RAID 5, and possibly RAID10/01 if desired.

That would be great.

Question: how does booting from fakeraid work with e.g. RAID5? Reading the
bootsector will work fine, but I don't see how grub can read its stage
files from a RAID5 array.

Bug#497110: improved dmraid support in D-I

On Fri, Sep 12, 2008 at 03:38:47PM EST, Frans Pop wrote:
> On Friday 12 September 2008, Luke Yelavich wrote:
> > So far, I have a patch to allow os-prober to find other OSs on dmraid
> > devices, but nothing for grub-installer yet, as I have been trying to
> > track down why grub-installer wasn't using os-prober's output for other
> > OSs.
>
> Main reason is that I'm not sure that, even though I'd expect os-prober to
> detect other OSes correctly, I doubt that grub-installer would create
> correct entries for them.
> If you can test that, then adding os-prober support is of course fine.
> As you can see in the history of the BR, that is the one part of your
> changes I was unsure about.

Right. I noticed some debconf templates related to sataraid in grub-installer, but I am not sure what those are for. To try and figure this one out, Iwas thinking of adding a set -x command right after at the point where os-prober's data is fetched, and follow things from there to see why it doesn't use os-prober's output.
> > Ok, I'll go into detail where necessary later in this mail, but I will
> > shortly file bugs against all d-i packages I've touched, with patches
> > attached.
>
> That's not needed. Giuseppe has already taken your changes from the diffs
> between Debian and Ubuntu sources and I already have commits to our SVN
> repository prepared based on those (with some mostly mostly minor changes
> and some additions).
>
> What would be useful is if you could check those, which is probably
> easiest to do after they've been committed.

I've downloaded the respecitve svn branches, so I'll keep an eye on them.

> > > Question: how does booting from fakeraid work with e.g. RAID5?
> > > Reading the bootsector will work fine, but I don't see how grub can
> > > read its stage files from a RAID5 array.
> >
> > As long as the fakeRAID BIOS loads, and grub uses BIOS calls, all will
> > work. The BIOS sets everything up so that grub simply sees things as an
> > ordinary disk. No different to how booting from raid0 on fakeraid
> > works.
>
> OK, so that is a case where having real hardware _does_ make a difference
> and an emulator without the BIOS support cannot be used for testing.

You could still test them in an emulator, however you can't boot from them.So you could have 3 disks for RAID 5, and a 4th with an install on it to boot from.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)

Bug#498846: marked as done (OOM if number of Choices and Choices-C are not equal)

Your message dated Sun, 14 Sep 2008 20:47:10 +0000
with message-id
and subject line Bug#498846: fixed in cdebconf 0.135
has caused the Debian Bug report #498846,
regarding OOM if number of Choices and Choices-C are not equal
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)

Bug#497110: improved dmraid support in D-I

On Friday 12 September 2008, Luke Yelavich wrote:
> On Fri, Sep 12, 2008 at 03:38:47PM EST, Frans Pop wrote:
> > On Friday 12 September 2008, Luke Yelavich wrote:
> > > Ok, I'll go into detail where necessary later in this mail, but I
> > > will shortly file bugs against all d-i packages I've touched, with
> > > patches attached.
> >
> > That's not needed. Giuseppe has already taken your changes from the
> > diffs between Debian and Ubuntu sources and I already have commits to
> > our SVN repository prepared based on those (with some mostly mostly
> > minor changes and some additions).
> >
> > What would be useful is if you could check those, which is probably
> > easiest to do after they've been committed.

The changes have now been committed as revisions r55850 to r55856. The
first commit is a semi-related bug fix. The second and last commits are
basically your work with a few minor cleanups. The other commits are
improvements by me created while I was testing the changes.
> I've downloaded the respecitve svn branches, so I'll keep an eye on
> them.

Bug#498845: marked as done (Causes unmatched Choices when first item in question is a divider)

Your message dated Mon, 22 Sep 2008 01:17:11 +0000
with message-id
and subject line Bug#498845: fixed in partman-base 126
has caused the Debian Bug report #498845,
regarding Causes unmatched Choices when first item in question is a divider
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)