Post subject: How do you rewrite the kernel if you chose use UEFI to start

Posted: Thu Sep 19, 2019 2:46 am

Joined: Fri Nov 23, 2018 10:53 pmPosts: 22

I am confues about this problem. Nowaday, I still use the BIOS to start my os. I meet so many problems, So I chose use UEFI to load this kernel. But, It isn't easy for me. I read so many books about the UEFI. Almost of they teach me how to use the edk2 to build the bootload.efi. I understand how to build the .efi file and use the qemu with ovmf.fd to start .efi file like bootload.efi. But, I don't know how to rewrite the kernel. we need to use the edk2 to compile our kernel? and I need to change the GDT IDT? I need to rewrite the thread? Can I still use the 0x8b000 to print char? The kernel may not be the kernel.bin, The kernel became kernel.efi???? I really want to know those basic problem. Thank you so much!

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Thu Sep 19, 2019 8:12 am

Member

Joined: Wed Aug 30, 2017 8:24 amPosts: 458

Since I dislike the (IMO) hacky way many higher-half kernels are linked together (i.e. with some parts linked at 1MB and some at 3GB, or -2GB in my case), I have always designed my kernel with a separate boot loader in mind. The kernel is a normal ELF executable that happens to be linked at -2GB, and the bootloader is whatever it needs to be. For BIOS boot, it is a multiboot executable linked at 1MB. For UEFI it is an EFI executable linked as PIC. Both load the kernel image, setup the environment my kernel needs, and transfer control there. The kernel itself needs not concern itself with the bootloaders too much, it merely needs to clear the temporary mappings from the page table (needed to enter long mode). And then the loaders are gone. The kernel does not know where it was loaded to (in PMem), except in the physical page allocator.

This is what I suggest everyone do. It is the most flexible possible, and if a new firmware standard comes about to address the shortcomings in UEFI, then that's just another loader for me.

As for graphics output: No, VGA compatibility mode is not available under UEFI. There are graphics output protocols you can use even after ExitBootServices(), though. If your kernel is flexible enough, it only needs the necessary pointers in its start parameter structure. Hardware text mode has some shortcomings, most of all that you have to trust the builtin fonts of the hardware. I suggest you switch to graphical mode (multiboot header option in legacy mode, or UEFI protocol in UEFI mode), and either add a font renderer to your kernel, or export the framebuffer and let userspace handle it. I put debug messages into the serial line. That is easier and almost always possible.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Thu Sep 19, 2019 10:31 am

Member

Joined: Fri Feb 17, 2017 4:01 pmPosts: 429Location: Ukraine, Bachmut

nullplan, how PE has become PIC? and the main thing why. but anyway, UEFI OS Loader does not need to be PIC, because it gets loaded by FW which is well aware of how to load PE images. so you can link it at whatever base you want, FW relocates it if needed.

for the OP. I think, the kernel should not change regarding booting at least. of course, you would want to take advantage of UEFI Runtime Services, - that would require changes, but it's another story.

you design your loader to comprise of two parts, say frontend and backend. the former is boot environment specific (FW specific) and the latter is not, the latter purely cares about deploying what the kernel expects. FE loads needed files, using services boot environments provide, processes interaction with user and prepares things for BE. then BE - in the UEFI, it would start roughly after ExitBootServices() - takes care of mapping building, loading (as executables) images - relocations, maybe something else.read the first chapters of the UEFI spec. to get knowledge, it really helps with understanding how to design your OS Loader.

and yet, graphics protocol is not available after ExitBootServices(), it is for Boot Services stage. there is just provision for the OS to get info on using framebuffer directly after exit BS. the mode is set, the info is exported, so the loader at the BE stage or kernel at the very early stage might draw into framebuffer directly.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Thu Sep 19, 2019 6:42 pm

Joined: Fri Nov 23, 2018 10:53 pmPosts: 22

nullplan wrote:

Since I dislike the (IMO) hacky way many higher-half kernels are linked together (i.e. with some parts linked at 1MB and some at 3GB, or -2GB in my case), I have always designed my kernel with a separate boot loader in mind. The kernel is a normal ELF executable that happens to be linked at -2GB, and the bootloader is whatever it needs to be. For BIOS boot, it is a multiboot executable linked at 1MB. For UEFI it is an EFI executable linked as PIC. Both load the kernel image, setup the environment my kernel needs, and transfer control there. The kernel itself needs not concern itself with the bootloaders too much, it merely needs to clear the temporary mappings from the page table (needed to enter long mode). And then the loaders are gone. The kernel does not know where it was loaded to (in PMem), except in the physical page allocator.

This is what I suggest everyone do. It is the most flexible possible, and if a new firmware standard comes about to address the shortcomings in UEFI, then that's just another loader for me.

As for graphics output: No, VGA compatibility mode is not available under UEFI. There are graphics output protocols you can use even after ExitBootServices(), though. If your kernel is flexible enough, it only needs the necessary pointers in its start parameter structure. Hardware text mode has some shortcomings, most of all that you have to trust the builtin fonts of the hardware. I suggest you switch to graphical mode (multiboot header option in legacy mode, or UEFI protocol in UEFI mode), and either add a font renderer to your kernel, or export the framebuffer and let userspace handle it. I put debug messages into the serial line. That is easier and almost always possible.

So, After ExitBootServices(), We can still use the UEFI service? Our kernel format isn't kernel.bin or kernel.efi?if I still want to use the UEFI service in Kernel.bin. How do I do?

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Thu Sep 19, 2019 10:30 pm

Member

Joined: Wed Aug 30, 2017 8:24 amPosts: 458

UEFI has boot services and runtime services. After ExitBootServices(), you can still use the runtime services. And some graphics protocols were set up to be runtime services.

As for the "how", you have to call SetVirtualAddressMap() after ExitBootServices(). Since I map all physical memory to just after the canonical break (i.e. 0xffff000000000000), I just tell UEFI about that part and gloss over the rest of virtual memory. You need to keep a pointer to the system table around. That requires that you don't corrupt it, either. Then just call whatever functions you want that are still accessible. I can't seem to find anything in the specs that allows you to call into the GOP functions, but also nothing that prohibits it.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Thu Sep 19, 2019 11:04 pm

Member

Joined: Fri Feb 17, 2017 4:01 pmPosts: 429Location: Ukraine, Bachmut

only Runtime Services functions are valid and guaranteed to work after ExitBootServices(), GOP functions won't work. you most probably won't even be able to call them, since the memory, where GOP instances code/data reside is not typed as Runtime Services code/data, this only thing already is enough to get that it can't be used after exit BS. but there is more: for example GOP functions almost certainly use BS functions, like pool functions etc. and in fact, the specification is clear on what you may use only framebuffer itself (and info about it taken from GOP) in the transient period after exit BS, not GOP. as well as only RS are legit to have been used at the OS runtime, given the latter preserved their memory, for which btw that set virtual mapping function is not necessary and moreover is mildly not recomended - the spec hints one is better off to establish 1:1 map and let RS run on addresses they were put by the FW. for example creating a process dedicated for this is a good way to accomplish this.but of course, you can try to check it out if it prohibits or not. it's a matter of just one hang to realize. or a few of them if you are persistent. -_-

only Runtime Services functions are valid and guaranteed to work after ExitBootServices(), GOP functions won't work. you most probably won't even be able to call them, since the memory, where GOP instances code/data reside is not typed as Runtime Services code/data, this only thing already is enough to get that it can't be used after exit BS. but there is more: for example GOP functions almost certainly use BS functions, like pool functions etc. and in fact, the specification is clear on what you may use only framebuffer itself (and info about it taken from GOP) in the transient period after exit BS, not GOP. as well as only RS are legit to have been used at the OS runtime, given the latter preserved their memory, for which btw that set virtual mapping function is not necessary and moreover is mildly not recomended - the spec hints one is better off to establish 1:1 map and let RS run on addresses they were put by the FW. for example creating a process dedicated for this is a good way to accomplish this.but of course, you can try to check it out if it prohibits or not. it's a matter of just one hang to realize. or a few of them if you are persistent. -_-

Could you tell me what version of the spec your looking at and what sections of that spec you found this information in? The latest UEFI spec says nothing about how SetVirtualAddressMap() is not recommended over something else. (It does indicate (only in section 1.2 as a side note) that the console support protocol "defines I/O protocols that handle input and output of text-based information intended for the system user while executing in the boot services environment," however.) For reference, I'm looking at chapter 8, but have searched the entire spec loking for "SetVirtualAddressMap" in particular, and found no such reference.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Fri Sep 20, 2019 4:59 pm

Member

Joined: Fri Feb 17, 2017 4:01 pmPosts: 429Location: Ukraine, Bachmut

Ethin wrote:

Could you tell me what version of the spec your looking at and what sections of that spec you found this information in? The latest UEFI spec says nothing about how SetVirtualAddressMap() is not recommended over something else. (It does indicate (only in section 1.2 as a side note) that the console support protocol "defines I/O protocols that handle input and output of text-based information intended for the system user while executing in the boot services environment," however.) For reference, I'm looking at chapter 8, but have searched the entire spec loking for "SetVirtualAddressMap" in particular, and found no such reference.

If you ask me about SetVirtualAddressMap() particularly, then the answer is it doesn't not recommend clearly, it just when you read and compare what one needs to do with and without calling that function, it becomes clear, that calling it is not worth of a try. Just read the description of SetVirtualAddressMap() and ConvertPointer() - when thinking on how many things should get fixed, it overwhelmes. it's a begging for troubles. also, there are passages like this, they leave a bit of confusion, the version 2.4.errata.b, section 2.3.6, page 39:

Quote:

In general, UEFI Configuration Tables loaded at boot time (e.g., SMBIOS table) can becontained in memory of type EfiRuntimeServicesData (recommended and the systemfirmware must not request a virtual mapping), EfiBootServicesdata,EfiACPIReclaimMemory or EfiACPIMemoryNVS. Tables loaded at runtime must becontained in memory of type EfiRuntimeServicesData (recommended) orEfiACPIMemoryNVS.

So, what does this mean? these tables are recommended to be typed as runtime data and firmware "must not request virtual mapping"... So when you will decide to still call SetVirtualAddressMap() in this case, the tables will get innaccessible?

I mean, you still can use runtime services without all this burden. just ensure that with enabled paging, addresses left 1:1 mapped. in every section for calling convention for every supported architecture, the specification says something like this:

Quote:

• The system address regions described by all the entries in the EFI memory map that have theEFI_MEMORY_RUNTIME bit set must be identity mapped as they were for the EFI bootenvironment.

and then adds this:

Quote:

If the OS Loader or OS used SetVirtualAddressMap() to relocate theruntime services in a virtual address space, then this condition does not have to be met.

For me it sounds like "man, why you would need that SetVirtualAddressMap() anyway?" it's my impression, given how horrendously broken that "fixing up" could go. Imagine some crappy implementation of FW in the runtime driver stores pointers, say in some internal structure, dynamically placed somewhere inside of its data. FW could fixup addresses by the PE mechanism, thus those, for which there is relocation information. these dynamically stored pointers might get overlooked and left unfixed easily, with all the consequences. But on the other hand, calling this function for real is not necessary, because you just can identity map all Runtime Services and be happy.

If you ask about where I got about GOP not being available after ExitBootServices(), then it is in many places. For example in the description of ExitBootServices():

Quote:

No further calls to boot service functions or EFI device-handle-based protocols may be used, and the boot services watchdog timer is disabled.

more than enough.

I remember reading somewhere a more obvious hint on not messing up with establishing non-identity mapping for Runtime Services, but I didn't find it now. Nevertheless, I think using it is not needed. you realize that you need to map every runtime typed region? then you need to rely on the FW fixing up every address in there. And, in the end of the day, it gives you absolutely nothing.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Sun Sep 22, 2019 8:38 am

Member

Joined: Thu Oct 13, 2016 4:55 pmPosts: 635

Hi,

nullplan's design is very good, that's what I do too. I also have a separate BIOS/Multiboot and UEFI loader, both capable of loading the kernel and providing exactly the same environment for it regardless to the firmware.

benji wrote:

So, After ExitBootServices(), We can still use the UEFI service? Our kernel format isn't kernel.bin or kernel.efi?if I still want to use the UEFI service in Kernel.bin. How do I do?

As said before, you can only use the ones which are specified as a "Run-Rime Service". For what format you choose for your kernel, is totally up to you, as it's parsed by your bootloader. You could use a.out, standard PE, UEFI PE or your own executable format if you'd like. The ABI is also totally up to you, the firmware doesn't care (because it's your bootloader that calls the entry point in your kernel, not the firmware).

To access UEFI Run-Time Services from your Kernel.bin, then you'll have to pass the UEFI System Table pointer to _start in your kernel, so that it can get the functions' addresses. After that, you are obliged to the UEFI ABI on calls but that's all. For example, the GNU efi implementation provides a function uefi_call_wrapper(), which assumes kernel native ABI interface on the input side (CDECL, SysV, fastcall, Pascal or whatever ABI you're compiling it to) and translates the parameters into a structure that fullfills the UEFI calling convention. On return, it also converts the return value according to the native ABI.

At a minimum, you'll need such a wrapper function in your kernel (unless your kernel is using the UEFI calling convention natively, very unlikely). Here's one implementation written for x86_64 in FASM Assembly. Here the actual call is handled in ueficall, and uefi_call_wrapper is a macro so that you can use it regardless which calling convention you prefer.You could write some more convenient functions too, like Print() which calls uefi_call_wrapper(ST->ConOut->OutputString) like GNU efi does, but that's not a must. Read more.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Mon Sep 23, 2019 11:57 pm

Joined: Fri Nov 23, 2018 10:53 pmPosts: 22

bzt wrote:

Hi,

nullplan's design is very good, that's what I do too. I also have a separate BIOS/Multiboot and UEFI loader, both capable of loading the kernel and providing exactly the same environment for it regardless to the firmware.

benji wrote:

So, After ExitBootServices(), We can still use the UEFI service? Our kernel format isn't kernel.bin or kernel.efi?if I still want to use the UEFI service in Kernel.bin. How do I do?

As said before, you can only use the ones which are specified as a "Run-Rime Service". For what format you choose for your kernel, is totally up to you, as it's parsed by your bootloader. You could use a.out, standard PE, UEFI PE or your own executable format if you'd like. The ABI is also totally up to you, the firmware doesn't care (because it's your bootloader that calls the entry point in your kernel, not the firmware).

To access UEFI Run-Time Services from your Kernel.bin, then you'll have to pass the UEFI System Table pointer to _start in your kernel, so that it can get the functions' addresses. After that, you are obliged to the UEFI ABI on calls but that's all. For example, the GNU efi implementation provides a function uefi_call_wrapper(), which assumes kernel native ABI interface on the input side (CDECL, SysV, fastcall, Pascal or whatever ABI you're compiling it to) and translates the parameters into a structure that fullfills the UEFI calling convention. On return, it also converts the return value according to the native ABI.

At a minimum, you'll need such a wrapper function in your kernel (unless your kernel is using the UEFI calling convention natively, very unlikely). Here's one implementation written for x86_64 in FASM Assembly. Here the actual call is handled in ueficall, and uefi_call_wrapper is a macro so that you can use it regardless which calling convention you prefer.You could write some more convenient functions too, like Print() which calls uefi_call_wrapper(ST->ConOut->OutputString) like GNU efi does, but that's not a must. Read more.

Cheers,bzt

thank you so much!!! But I still have a small problem that is How to open the interruption when you using the uefi to boot your kernel. For example. In BIOS, We can register the IDT(interruption description table) and the CPU can fin d the interrupt handlers address in IDT. Therefore, CPU can run the interrupt handler. Can we still use that way to boot our kernel? The UEFI may includes a new way to find the interrupt handlers? if We use the UEFI, can we still use the interrupt handlers in kernel.bin?

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Tue Sep 24, 2019 12:27 am

Joined: Fri Nov 23, 2018 10:53 pmPosts: 22

benji wrote:

bzt wrote:

Hi,

nullplan's design is very good, that's what I do too. I also have a separate BIOS/Multiboot and UEFI loader, both capable of loading the kernel and providing exactly the same environment for it regardless to the firmware.

benji wrote:

So, After ExitBootServices(), We can still use the UEFI service? Our kernel format isn't kernel.bin or kernel.efi?if I still want to use the UEFI service in Kernel.bin. How do I do?

As said before, you can only use the ones which are specified as a "Run-Rime Service". For what format you choose for your kernel, is totally up to you, as it's parsed by your bootloader. You could use a.out, standard PE, UEFI PE or your own executable format if you'd like. The ABI is also totally up to you, the firmware doesn't care (because it's your bootloader that calls the entry point in your kernel, not the firmware).

To access UEFI Run-Time Services from your Kernel.bin, then you'll have to pass the UEFI System Table pointer to _start in your kernel, so that it can get the functions' addresses. After that, you are obliged to the UEFI ABI on calls but that's all. For example, the GNU efi implementation provides a function uefi_call_wrapper(), which assumes kernel native ABI interface on the input side (CDECL, SysV, fastcall, Pascal or whatever ABI you're compiling it to) and translates the parameters into a structure that fullfills the UEFI calling convention. On return, it also converts the return value according to the native ABI.

At a minimum, you'll need such a wrapper function in your kernel (unless your kernel is using the UEFI calling convention natively, very unlikely). Here's one implementation written for x86_64 in FASM Assembly. Here the actual call is handled in ueficall, and uefi_call_wrapper is a macro so that you can use it regardless which calling convention you prefer.You could write some more convenient functions too, like Print() which calls uefi_call_wrapper(ST->ConOut->OutputString) like GNU efi does, but that's not a must. Read more.

Cheers,bzt

thank you so much!!! But I still have a small problem that is How to open the interruption when you using the uefi to boot your kernel. For example. In BIOS, We can register the IDT(interruption description table) and the CPU can fin d the interrupt handlers address in IDT. Therefore, CPU can run the interrupt handler. Can we still use that way to boot our kernel? The UEFI may includes a new way to find the interrupt handlers? if We use the UEFI, can we still use the interrupt handlers in kernel.bin?

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Tue Sep 24, 2019 5:57 am

Member

Joined: Thu Oct 13, 2016 4:55 pmPosts: 635

benji wrote:

thank you so much!!! But I still have a small problem that is How to open the interruption when you using the uefi to boot your kernel. For example. In BIOS, We can register the IDT(interruption description table) and the CPU can fin d the interrupt handlers address in IDT. Therefore, CPU can run the interrupt handler. Can we still use that way to boot our kernel? The UEFI may includes a new way to find the interrupt handlers?

Okay I see your confusion. UEFI does not use interrupts like BIOS does. BIOS hooks itself in the IVT (real mode IDT) to provide services, which usually got lost as soon as a protected/long mode kernel sets up its own IDT. Therefore UEFI does not do that, it provides a separate EFI System Table pointer with the services' function pointers. UEFI does not provide interrupt handlers at all, as it keeps the IDT in the realm of the CPU (see sidt / lidt instructions). As such, you can't boot your kernel using interrupts under UEFI, there you have to make UEFI firmware calls (see the EFI System Table and the aforementioned uefi_call_wrapper() function).

benji wrote:

if We use the UEFI, can we still use the interrupt handlers in kernel.bin?

Yes of course, but only interrupt handlers that kernel.bin sets up for itself. Actually 99% of the kernels out there sets up their own IDT as soon as possible, so BIOS or not, they start using their own handlers as soon as possible.

benji wrote:

Could you show a simple OS code using UEFI?

Check out my MIT-licensed Open Source boot loader, BOOTBOOT. Actually it's not a single boot loader, but a Boot Protocol (see the PDF) with several small implementations, one for BIOS, one for UEFI etc. The BOOTBOOT loader has exactly the opposite philosophy than GRUB, it is not a Grand Unified Loader, rather a set of simple and small loaders which provide the same boot environment on different platforms.

There's a minimal "hello world" kernel which can be compiled for x86_64 and AArch64, and my loader can boot that kernel under BIOS and UEFI as well. You can use that example source as a skeleton for your own kernel if you want.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Wed Sep 25, 2019 6:49 pm

Joined: Fri Nov 23, 2018 10:53 pmPosts: 22

bzt wrote:

benji wrote:

thank you so much!!! But I still have a small problem that is How to open the interruption when you using the uefi to boot your kernel. For example. In BIOS, We can register the IDT(interruption description table) and the CPU can fin d the interrupt handlers address in IDT. Therefore, CPU can run the interrupt handler. Can we still use that way to boot our kernel? The UEFI may includes a new way to find the interrupt handlers?

Okay I see your confusion. UEFI does not use interrupts like BIOS does. BIOS hooks itself in the IVT (real mode IDT) to provide services, which usually got lost as soon as a protected/long mode kernel sets up its own IDT. Therefore UEFI does not do that, it provides a separate EFI System Table pointer with the services' function pointers. UEFI does not provide interrupt handlers at all, as it keeps the IDT in the realm of the CPU (see sidt / lidt instructions). As such, you can't boot your kernel using interrupts under UEFI, there you have to make UEFI firmware calls (see the EFI System Table and the aforementioned uefi_call_wrapper() function).

benji wrote:

if We use the UEFI, can we still use the interrupt handlers in kernel.bin?

Yes of course, but only interrupt handlers that kernel.bin sets up for itself. Actually 99% of the kernels out there sets up their own IDT as soon as possible, so BIOS or not, they start using their own handlers as soon as possible.

benji wrote:

Could you show a simple OS code using UEFI?

Check out my MIT-licensed Open Source boot loader, BOOTBOOT. Actually it's not a single boot loader, but a Boot Protocol (see the PDF) with several small implementations, one for BIOS, one for UEFI etc. The BOOTBOOT loader has exactly the opposite philosophy than GRUB, it is not a Grand Unified Loader, rather a set of simple and small loaders which provide the same boot environment on different platforms.

There's a minimal "hello world" kernel which can be compiled for x86_64 and AArch64, and my loader can boot that kernel under BIOS and UEFI as well. You can use that example source as a skeleton for your own kernel if you want.

Post subject: Re: How do you rewrite the kernel if you chose use UEFI to s

Posted: Sun Nov 03, 2019 10:18 pm

Joined: Mon Jun 13, 2016 2:25 amPosts: 24Location: Sydney

benji wrote:

The kernel may not be the kernel.bin, The kernel became kernel.efi???? I really want to know those basic problem. Thank you so much!

Apologies for resurrecting a dead thread, I figure it's worth a reply in case this helps someone.No one has addressed the fact that you shouldn't be compiling your kernel into the UEFI bootloader. This statement suggests that OP is a little confused regarding the bootloader's purpose and how it differs from the kernel itself. UEFI is not for writing a kernel, it's for writing a bootloader to load the kernel. The bootloader's job is to set up the initial environment for loading and executing the kernel. While the kernel is highly coupled to the bootloader, they are separate programs entirely. The UEFI bootloader should be compiled as an EFI executable, which then loads the separate kernel executable in whatever binary format it is in.Regarding OP's original question of what you need to rewrite in your kernel: This is a really pedantic answer, but... nothing. Maybe. Most online tutorials suggest using GRUB and writing your kernel to adhere to the Multiboot standard. An advantage of doing this is that it would theoretically be compatible with any bootloader implementing this standard. If you roll your own bootloader you're going to need to decide for yourself what state your kernel expects the environment to be in.If you're having trouble with writing your kernel, UEFI won't offer you much help. Writing your own bootloader in UEFI will not simplify any part of the kernel development process.

Who is online

Users browsing this forum: Google [Bot] and 15 guests

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot post attachments in this forum