Xen PV domU - How to run pfSense in paravirtulised mode

Abstract
The main purpose of this topic is to provide a simple way and share patches to build a pfSense DomU in paravirtualised mode for Xen Hypervisor.

Why I'm doing that : For the challenge and because my personnal Xen Hypervisor Host does'nt have any Virtualisation set built in. (Atom)

Good to know before building
Xen Kernel for domU in RELENG_8_1 (FreeBSD 8.1) is not really stable, I got a lot of panic and a lot of feature where not present at this stage of freeze in Xen code.

I started in two way :

Try to backport Xen kernel code the the RELENG_8_1 kernel, hard work and I'm not familliar with code versionning for FreeBSD so I give up.

Build pfSense based on RELENG_8_2 (Xen kernel is more complete at this stage).

Building your own pfSense domU

1 - Before anything else, you must install a freebsd8.1 from scrach (I run it on another VM)
2 - Follow the instruction to setup your own pfSense build scratchbox ( http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso )
3 - Make sure that the building process is working by following the instruction in the wiki
4 - Now we need to make some custom things in order to make it build for RELENG_8_2

Customisation 1 : Make the Kernel Xen Aware
Create your own KERNCONF file to add XEN support (and keeping pfsense stuff in it). For this tack I took the pfsense_wrap.8.i386 file and the XEN one from freebsd sources and mixed them.

Warning XEN is including PAE so there some device to remove (scsi stuff, and some others) because they stop when compiling complaining of memory. In my file I remove all the devices including wireless because I don't need them at the moment. Feel free to custom yourself if you need specific device with PCIpasstrough from Xen (excep the need of wireless device, I don't see others to keep in domU).

See below for my pfSense_wrap.8.XEN file.

To use your custom file, override the existing one (make a backup before) in /home/pfsense/tools/builder_script/conf/pfSense_wrap.8.i386

Next you have to modify /home/pfsense/tools/builder_scripts/conf/src.conf.embedded.8 to add the module we don't want to build (using makeoption WITHOUT_MODULES in KERNCONF seems not working fine with RELENG_8_2). So add :

Thanks for this very useful post! I am a total newbie in the field of pfSense and/or freeBSD, but your post really helped me to get started.

I want to implement pfSense in a domU on my home server, which has a simple Pentium IV processor without virtualization support. I overcame quite a few problems but now I seem to be stuck. I will start with give a short overview of some of the problems I solved, once I manage (if I manage…) to complete the procedure I will post my solutions in more detail.
Disclaimer: I hardly know what I am doing. My solutions need to be reviewed by someone a bit more exprienced in this matter before it can be called 'reliable'!

What did I encounter?

Even with the modified patches that you provided, I encountered quite a few patches that didn't work. I analyzed and modified those patches by hand and I managed to make all patches work without modifying any of the code itself. See attachments.

One patch turned out to lead to a compile error, I modified the code so it would work. (This might need some review from someone who, unlike me, would actually know what he is doing)
Specifically: in if_rum_pr_144642.diff I changed "return 0;" to "return;". This solved the compile error.

Three pfPorts did not build due to compile errors. I solved two. So far I did not yet manage to install php52-pfSense-module; however, this does not seem to be related to the issue below.

One important addition to the procedure above is that in my pfSense_wrap.8.i386 file (see attachment), I explicitly excluded everything that I could encounter that might be related to USB, SCSI or wireless support. To achieve this I used the 'nodevice' and 'nooptions'. My purpose is that no USB (or SCSI or wireless) related materials will be built - not directly into the kernel, and not as modules. However, as can be seen in the problems I encounter, this does not work as I expected.

So where does it go wrong?
I start the procedure using the /usr/home/pfsense/toold/builder_scripts/build_nano.sh script. I get the same result when I start the procedure from the menu, as described above.

Everything seems to run fine up to the time that it starts buidling the kernel. At "stage 3.1: making dependencies", the following error occurs:

/usr/pfSensesrc/src/sys/dev/usb/net/if_udav.c:72:21: error: usbdevs.h: No such file or directory

I have attached the full output of the build_nano.sh command; normal (build_nano_output-normal.txt) and after uncommenting 'set -x' in the build_nano.sh script (build_nano_output-debug.txt).

To get sticked I think there is some work to be done by other member. I fell alone with this topic so I give up for the moment (other things to do like my wedding for example).

I think the best way is to use the appliance part of pfSense build tools, we also need a lot of patches from last kernel and xen devices (the xn network device for example that need some improvement if we need to manage VLAN inside pfSense or QOS with bandwidth tweaks).

Feel free to try and Improve the receipe, if this topic is moving on, I will do my best to continue my work.