(In reply to comment #3)
> Chris, this is a royal pain for me to do, it seems very unlikely that it's the
> same issue...
I'm not sure why, on either count. If you can just boot without loading the PV-on-HVM drivers, install the updated kernel, and then boot again with the PV-on-HVM drivers, that should show it. Also, the call trace here looks very similar to the one in 477005; while that one was about our -debug variant, the fact remains that the debug code is pointing out a potential problem that could hit the non-debug variant.
It still may be a different issue, but it should be a relatively easy test to see if it works with the updated kernels.
Chris Lalancette

This is a post I made earlier to xen-discuss@opensolaris.org, reposted here as per John Levon's request
================================
I'm using RHEL5.3 x86_64 as dom0 and Centos 5.3 x86_64 HVM domU, with
Xen 3.3.1. By default PV drivers are LOADED, but not USED. hda is
still handled by ata_piix, and eth0 handled by 8139cp. xen-vbd is
loaded, but it does not have any devices (because hda is already
handled by ata_piix). xen-net is loaded and got eth1 with the same MAC
address as eth0.
During boot, you can see that xen-vbd tries to grab hda but cannot
(I'll get to this later). This works fine in Linux, domU can continue
to boot. To get domU to actually USE the PV drivers for hda and eth0,
some steps are required : http://pastebin.com/fb6fe631
The point here is that on Linux dom0, whatever driver handles hda,
domU can continue to work.
Now here comes the funny part. In opensolaris dom0, the process of
xen-vbd trying to grab hda actually cause KERNEL PANIC.
So here's what I did to get it working on opensolaris dom0:
- do a zvol-backed fresh-install of Centos 5.3 on opensolaris dom0. I
got kernel-2.6.18-128.el5. It panics on the first reboot.
- export the zvol with iscsi, mount on linux, move
/lib/modules/2.6.18-128.el5/kernel/drivers/xenpv_hvm/blkfront/xen-vbd.ko
out of the way (I simply rename it to xen-vbd.ko.disabled)
- startup domU again on opensolaris
- edit /etc/modprobe.conf, add "alias scsi_hostadapter2 xen-vbd". This
will ensure xen-vbd is included on initrd later.
- yum -y install kernel. I'll get kernel-2.6.18-128.1.6.el5, and
during the installation process it also generates
initrd-2.6.18-128.1.6.el5.img which contains xen-vbd.ko.
- edit /boot/grub/menu.lst to look like this
title CentOS (2.6.18-128.1.6.el5)
root (hd0,0)
kernel /boot/vmlinuz-2.6.18-128.1.6.el5 ro root=LABEL=/ ide0=noprobe
initrd /boot/initrd-2.6.18-128.1.6.el5.img
the part "ide0=noprobe" tells ata_piix (or is it libata?) to NOT probe
hda and hdb.
- reboot
On the reboot process you can see that xen-vbd now handles hda. Note
that hdc (the cdrom) is still handled by ide-cd. As a bonus, xen-net
now handles eth0.
[root@localhost ~]# ls -lad /sys/block/hd*/device
/sys/block/hd*/device/driver/module
/sys/class/net/eth*/device/driver/module
lrwxrwxrwx 1 root root 0 May 8 03:57 /sys/block/hda/device ->
../../devices/xen/vbd-768
lrwxrwxrwx 1 root root 0 May 8 04:19
/sys/block/hda/device/driver/module -> ../../../../module/xen_vbd
lrwxrwxrwx 1 root root 0 May 8 03:57 /sys/block/hdc/device ->
../../devices/pci0000:00/0000:00:01.1/ide1/1.0
lrwxrwxrwx 1 root root 0 May 8 04:19
/sys/block/hdc/device/driver/module -> ../../../../module/ide_cd
lrwxrwxrwx 1 root root 0 May 8 04:17
/sys/class/net/eth0/device/driver/module ->
../../../../module/xen_vnif
lrwxrwxrwx 1 root root 0 May 8 04:19
/sys/class/net/eth1/device/driver/module -> ../../../../module/8139cp