If LPAR has a repository node ios.ata.region0.access (value doesn't matter) then the access rights check never fails. After System Manager sets ATA keys it removes this repository node from LPAR 1. If we add this repository node again or patch System Manager so it's not removed then we will be able to access all storage regions of all storage devices.

The commands can be used with HV call lv1_storage_send_device_command.

However, before a command is executed HV does bit manipulation with it and checks it against the value of repository node ss.laid or also called LPAR Authority ID. If this test fails then the command is NOT executed.

Here is the list of commands supported by FLASH StarShip 2 storage device.

The commands can be used with HV call lv1_storage_send_device_command.

However, before a command is executed HV does bit manipulation with it and checks it against the value of repository node ss.laid or also called LPAR Authority ID. If this test fails then the command is NOT executed.

It seems that TOC 0xC0000 and TOC 0x7C0000 contain the same files but from different SDK versions.

TOC 0xC0000 is SDK version 3.41 and TOC 0x7C0000 is SDK version 3.30 (look at the content of files sdk_version).

I guess it's because when i bought my PS 3 Slim it had Firmware 3.30 and i updated it to 3.41 for PSGroove.

TOC on /dev/rflash1 is used by HV Processes to locate files and load them into memory, e.g. SPU modules. E.g. Process 6 loads spu_utoken_processor.self to decrypt and verify user tokens or SPL which runs in Process 5 loads spp_verifier.self from there in order to decrypt and verify profile files. And Update Manager stores e.g. there files.

The commands can be used with HV call lv1_storage_send_device_command.

However, before a command is executed HV does bit manipulation with it and checks it against the value of repository node ss.laid or also called LPAR Authority ID. If this test fails then the command is NOT executed.

Each storage device has a table of size 16 to store incomming and pending Storage Requests

ENCDEC storage device has a table of size 32 to store incomming and pending Storage Requests

When a new Storage Request is submitted e.g. by HV call lv1_storage_read or lv1_storage_write, the table is scanned for a free slot, if there are no pending Storage Requests then the Storage Request is executed immediately

When a Storage Request is completed, the finished Storage Reuqest is passed to function storage_device_async_request_complete and the table of Storage Requests is scanned again for the next pending Storage Request which will be executed

There are 2 types of Storage Requests: Read/Write (1) and Device Command (2).

Read and Write Storage Requests use the same HV function of a Storage Device to enqueue the request. Before Write Storage Request is inserted into the Request Table of a Storage Device, the flags parameter passed e.g. in lv1_storage_read or lv1_storage_write is ored with 0x8. That is how HV differentiates between Read and Write Storage Requests.

Write request is first passed to ENCDEC device for encryption. When ENCDEC device is done, it calls a callback and passes the encrypted data to the callback. The callback writes the encrypted data with ATA WriteDMAExt command to HDD.

When a storage device request is processed by HV, Storage Subsystem checks if cryptography is enabled for the storage device.

HV checks 1 byte of data owned by the storage device and when the value of this flag is not 0 then it uses encryption/decryption.

By setting this flag to 0 at runtime, encryption/decryption of storage devices can be disabled at runtime.

We could patch lv1.self so that encryption/decryption of storage devices is disabled permanently.

The SPEType1 and SPEType2 objects cannot be created when isolation mode is disabled. The right most bit of repository node sys.lv1.iso_enbl is checked and when it's not 1 then the SPEType1 and SPEType2 objects cannot be created. In LPAR 1, this check succeedes always. Only in LPARs different from 1, the repository node sys.lv1.iso_enbl is checked.

The value of shadow_addr that is returned by lv1_construct_logical_spe is a LPAR start address of this area and it cannot be accessed until it's mapped in the HTAB.

The SPE Register Shadow Area may be mapped only with read-only page protection or else HV call lv1_insert_htab_entry fails. I tested it with PSGroove and could map the whole memory range and read it after i constructed SPE of type 1 with lv1_construct_logical_spe.

The shadow_addr is also returned by syscall_10040 (that creates SPE of type 1) but it returns already mapped Process address so HV Processes do not have to map it in HTAB.

When an isoated SPU is done, HV Processes checks the value at offset 0x30 to determine if the SPU execution was successfull or not.

GameOS checks also the value at offset 0x30 in the SPE Shadow Area.

When GameOS creates SPE of type 1 then it maps only SPE Register Shadow Area into it's address space.

Physical/Virtual memory address of an isolation module that should be loaded by metldr is written into SPU register SPU_In_Mbox. The SPU register SPU_In_Mbox is 32bit, so 64bit memory address is written in 2 steps.

The System Manager of GameOS sends the path to lv2_kernel.self to SLL (Secure LPAR Loader) and SLL loads it from FLASH device file /dev/rflash1

I stored a new lv2_kernel.self on FLASH directly by writing FLASH from GameOS. It't risky but if you know what you are doing then it's safe. I warned you guys. You could brick your PS3.

Then i added a new TOC entry to FLASH device which points to the new lv2_kernel.self

I patched the path to lv2_kernel.self in the System Manager of GameOS so it points to my new GameOS kernel (You need HV rights to do it)

Then i rebooted GameOS without rebooting HV, so the patched file path should not change

This method has the advantage that when the new lv2_kernel.self won't work you can just reboot HV and it will load the original lv2_kernel.self again

lv2_kernel.self can be also loaded from GameOS dev_flash. For that, you have to change the path to lv2_kernel.self in default.spp from /flh/os/lv2_kernel.self to /local_sys0/lv2_kernel.self and store lv2_kernel.self on dev_flash.

Processes 3, 5 and 6 provide services (functions) to other Processes through sockets (something like RPC).

A service is identified by a function ID.

Each process has a hash table which maps a function ID to socket port ID.

Services (functions) can be further differentiated by a packet ID.

To request a service, a Process sends a packet with specified function and packet ID to the Process that provides the service.

A process that provides a service (function) has a table of objects which handle different packet IDs.

Services are synchronous, a client sends a request and waits for a response.

If a Process requests a service that is located in the same Process then the service is called directly and sockets are not used !!! (e.g. SLL requests from DM creating VUART port during GameOS loading, SLL and DM are in the same Process, so SLL calls DM directly)

This method is used e.g. in all HV calls which create any kind of memory regions, e.g. lv1_allocate_memory, lv1_map_htab, lv1_undocumented_function_114, lv1_construct_logical_spe, lv1_map_device_mmio_region or syscall 0x10040.

GameOS allocates nearly all physical memory of PS3 for itself !!! That is why new HV calls lv1_allocate_memory with large memory region sizes will fail.

So when someone wants a large piece of physical memory, he can borrow it from GameOS's LPAR memory region that starts at 0x700020000000. It can be used for example to send update packages to Update Manager which are very large.

Here is the list of physical memory regions of GameOS i found in HV 3.41:

Here is a short description of the method i used to exploit HV from GameOS 3.15 and 3.41.

First i used the Geohot's method to create a dangling HTAB entry.

Making memory glitch work on GameOS was the largest of my obstacles but i solved it and i'm able to create a dangling HTAB entry from GameOS within 1-3 minutes.

Then i created many Direct Map Memory Region objects of size 0 with HV call lv1_undocumented_function_114 and checked if they are within the page to which the dangling HTAB entry points to.

When i found one such Direct Map Memory Region object i patched the size of this object to 0x1000. Then i pointed this memory region object to the code of HV call lv1_undocumented_function_114 and patched 4 bytes in this HV call which allows me to create any Direct Map Memory Region objects without any restrictions.

Function LPAR_construct_direct_mapping_mem_region which is used by HV call lv1_undocumented_function_114 has a parameter (register %r9) and when this parameter is not 0 then HV will allow you to create any Direct Map Memory Region objects without restrictions, but unfortunately the HV call lv1_undocumented_function_114 passes 0 in this parameter, so i just patched it.

Then i mapped whole HV memory range with the patched HV call lv1_undocumented_function_114 into the address space of GameOS.

And now you have read/write access to the whole HV.

$ONY could fix this exploit by disallowing creating of Direct Map Memory Region objects of size 0, but i know tons of other HV C++ classes which will allow me to exploit the HV in a similar way, so it wouldn't bring $ONY anything :-) And they have to change member variable offsets in those objects to make sure that i cannot patch them easily :-)

Event Notfication is used e.g. to notify a LPAR about some event, e.g. device interrupt or notify a LPAR about destruction of another LPAR.

For example Process 9 is notified through Event Notification when LPAR 2 is destructed.

During LPAR construction, Process 9 creates an Outlet object with syscall 0x1001A and then passes the outlet ID to the syscall 0x10009 that constructs the LINUX LPAR. In this way Process 9 is notified when LINUX LPAR is destructed.

At address 0x38(LPAR ptr) + 0x150 is stored the physical address of LPAR's VUART IRQ Bitmap that was passed to lv1_configure_virtual_uart_irq

When a VUART interrupt is generated by HV then first the VUART IRQ Bitmap owned by HV is updated and then this bitmap is copied to LPAR's VUART IRQ Bitmap, so VUART IRQ Bitmap is stored twice, once in HV and once in LPAR, just like IRQ State Bitmap.

VUART IRQ Bitmap is not allowed to cross page boundary of LPAR memory region where it is stored. HV checks it and makes sure that it doesn't happen.

GameOS 3.41 VUART IRQ bitmap is at address 0x80000000003556E8 and of size 32 bytes (256 bits, each bit corresponds to a VUART port).

SYSCON MMIO registers can be accessed on Linux with a driver using lv1_undocumented_function_114, e.g. ps3sbmmio.
Use ps3sbmmio device driver carefully, an access at some addresses could shutdown your PS3.

First, the Hypervisor reads a word from address 0x2400008E000, ors it with 0xFFFFFFFD and writes the value back.

Then, the Hypervisor reads a word from address 0x2400008E004 and tests if bit 0x2 is set or not. The bit 0x2 should be not 0 or else the Hypervisor panics.

After that, the Hypervisor reads a word at address 0x2400008CFF0 and 0x2400008DFF4. If there is a new packet pending from SYSCON, then the (value + 1) at 0x2400008CFF0 should be equal the value at 0x2400008DFF4.

The Hypervisor reads the header of the packet beginning at the address 0x2400008C000.

The header is read with 4 word transfers by the Hypervisor.

The byte at offset 1 in the packet header must be 1 or else the Hypervisor discards the packet as invalid.

The Hypervisor calculates the checksum of the packet header and checks it with the checksum stored in the header. If they don't match then the Hypervisor discards the packet.

The Hypervisor reads the body of the packet beginning at the address 0x2400008C010.

The header and the body of the received packet can be read as many times as you want !!! They remain until next SYSCON packet is received

which gives us the possibility to communicate with SYSCON on Linux easily :)

Value of the loader parameter "sys.hw.config" controls if Gelic WLAN is enabled or not.

Value of the loader parameter "sys.hw.config" is stored in the repository node "sys.hw.config" too.

If bit 0x40000 is set then LV1 allows using Gelic WLAN interface from LV2.

Value on my PS3 slim 0x4e00ffff0a03bc3c with Gelic WLAN interface disabled. As you can see, the Gelic WLAN interface is disabled and LV1 doesn't allow using of LV1 calls 196 and 195. It returns LV1_CONDITION_NOT_SATISFIED.

GameOS checks bit 0x40000 of the repository node "sys.hw.config" during network initialization and if it's set then LV2 initializes Gelic WLAN interface.

Check your "sys.hw.config" repository node and if bit 0x40000 is set then you are a lucky owner of a PS3 model with the old WLAN interface.

Linux kernel doesn't use Gelic Device Control Interface like GameOS does it.
To get WLAN working on Linux booted with GameOS rights, we have to disable
Gelic Device Control Interface first because it's enabled for GameOS by default.

The value of repository node "ios.net.eurus.lpar" controls access to Gelic Device Control Interface.
It's a bitmap. The position of a bit corresponds to LPAR id. During GameOS booting, HV process 9 (System Manager) sets bit at postion 2 to 1 which means enable Gelic Device Control Interface for LPAR 2.

To disable Gelic Device Control Interface on Linux, first unload Gelic device driver, then set
value of repository node "ios.net.eurus.lpar" to 0 and load Gelic device driver again. After that WLAN should work again but only on FATs.

For PS3 Slim we need a new Linux Gelic device driver which uses Gelic Device Control Interface directly.

Make sure you set multicast address filter properly or else you won't receive broadcast packets !!!

Bulk transfers returned by the host controller which do not contain any data have size of 0x10 bytes else transfers contain valid Ethernet frame. All 802.11 related data is stripped by the WLAN Gelic device.

Make sure you set right MAC address with command 0x115b else device won't be able to receive packets destined to its own MAC address !!!

ps3_jupiter.ko is the common part of STA and AP mode. It implements a command interface to WLAN Gelic device and disptaches events to STA and AP drivers.

ps3_jupiter_sta.ko is a STA mode implementation.

ps3_jupiter_ap.ko is a AP mode implementation.

Simple scanning works already in STA mode (try it out with iwlist scan)

Packet reception works

Packet transmission works

WPA/WPA2 fully working and usable with wpa_supplicant

Finally, after several weeks of hard programming and reversing, the WLAN driver ps3_jupiter_sta achieved the milestone where i can use it with WPA2 :) I actually use it currently with WPA2 on my PS3 slim. It works damn !!! Try it out and report bugs and problems to me.

Found the place in LV1 where LV1 sets IO page size for GART memory mapping. We could patch it and set to 4KB. That would make a lot of things easier for RSX developers on Linux.

1MB pages make RSX driver for Linux hard to implement because allocating 1Mb contiguous memory chunk on Linux is very very hard especially on a system with only 256MB and which was running for some time.

LV1 supports 16 contexts simultaneously.

LV1 has an array of context pointers.

Each context has an index and a handle. The handle is derived from the index of the context.

Decision: create new DRM driver in order to learn how DRM framework in Linux kernel works and because we have to use LV1 calls to access RSX (and because it's a lot more fun to do it on my own). But use nouveau as an example for DRM driver. Maybe i should better use radeon DRM driver as an example beacuse it seems to be better designed and implemnted !!!

The driver is very low level and allows direct access to almost all RSX funtions, e.g. FIFO buffer, to achieve maximum performance.

All data buffers, e.g. vertices and textures, are managed by DRM framework (Linux kernel). To avoid copying from user to kernel space, the buffers will be mmaped into user space.

Provides an interface to manage graphic objects in VRAM.

Use TTM or GEM ??? TTM is used by radeon and nouvea drivers, so i guess we could use it too. GEM is for Intel chips.

GART memory region is a memory region in System Memory but accessible by RSX through GART [7].

GameOS calls it Main Memory.

Problem: lv1_gpu_context_iomap supports ONLY 1MB and 64kB pages

Size of system memory objects mapped into GPU address space should be either multiple of 1MB which means wasting lots of RAM and we don't have enough of it anyways. This solution is NOT suitable.

Or place several GART memory objects into 1 MB page and map it. That would mean we have to use memory manager for each 1MB page.

That means, we have to allocate 1MB page even if user requested a smaller memory region. Then initialize a heap manager for this 1MB page and return ONLY requested size. The following requests for GART memory regions can be satisfied from the previously allocated 1MB pages which still have enough free memory.

FIFO command buffer is an example of a GART memory object which has to be mapped into GPU address space with lv1_gpu_context_iomap before it can be used by RSX.

User allocates FIFO command buffer in GART address space, maps it into user space, write commands into it and then pushes it to DRM driver which maps it into RSX address space and CALLs it.

TTM: TTM_PL_FLAG_TT for GART memory

GameOS applications using GCM library map GART memory beginning at offset 0x10000000 or 0x20000000, just after where the whole VRAM is mapped.

Don't use kmalloc for this type of memory. Use __get_free_pages and mark pages with flag VM_RESERVED before exporting it to user-space else they can be swapped out.

TTM uses page-wise allocation for buffers. The buffers are contiguous ONLY in a single page. That has a huge advantage over allocating 1MB contiguous memory blocks in kernel space. It's far easier to allocate a single page in Linux kernel than 1MB memory chunk, especially on PS3 arch which has only 256MB.

libdrm uses handles which are returned by DRM kernel driver when a new memory object is created. The handle is passed to mmap syscall as parameter offset. DRM driver looks up the handle and identifies the appropriate memory object which is mapped into user-space then.

Nouveau driver uses TTM framework to map memory objects into user-space. TTM doesn't map all pages owned by the memory object at once but installs VM operation fault which maps single pages on demand. It makes sense because user application rarely accesses all pages of the mapped memory object at once.

To map memory objects located in VRAM we have to map it into kernel space first with ioremap.

By user applications submitted FIFO command buffers are mapped by DRM driver into RSX address space first and then executed with CALL command.

Problem: All references to graphics objects contained in FIFO command buffers must be expressed in RSX address space. How does user space know the right offsets of the referenced objects ???

To solve the above problem, Nouveau driver uses relocations which are submitted to DRM driver together with FIFO command buffers. The DRM driver applies the specified relocations before executing the FIFO command buffer. See nouveau_gem.c:nouveau_gem_pushbuf_reloc_apply.

Relocations contain memory object handles which they apply to. The DRM driver looks up the memory object by its handle and the memory objects contain GPU address space offsets.

Direct access to video RAM from kernel is very very slow but some of frame buffer functions in Nouvea driver are hardware accelerated. We could do it the same way on Linux and get a hardware accelerated frame buffer this way. Not sure why ps3fb authors didn't add hardware acceleration to frame buffer. The reason why it was not implemnted in ps3fb is because LV1 doesn't create 2D graphic objects needed for 2D hardware acceleration.

I uploaded here a test kernel module and a test user application: [12] and [13]

I used a similar technique for mapping GPU resources into user-space like Linux kernel DRM drivers do it, e.g. Nouveau. But of course everything is very simplified in comparison with Nouveau driver. All GPU resources are mapped to user-space with mmap and there is no data copying between user and kernel space, for performance reasons. Mapping GPU resources into user-space like this is more flexible than IOCTLs.

The purpose of the kernel module and the user application is to test how RSX works, to test FIFO commands and other stuff i reversed from Lv2. It's NOT for end users.

Before loading the kernel module make sure ps3vram kernel module is NOT loaded.

I used 64kB IO pages for GPU context. 4kB IO page size would be definitely a lot better for that we have to patch LV1. I will add this patch to my ps3mfw tasks for LV1.

Just load the kernel module and then run the user application.

The user application maps all context resources and executes some simple FIFO commands, like JMP or SET REF.

I will add more examples later.

By default, the kernel module allocates 8MB VRAM, 64kB FIFO and 1MB GART memory. You can change it by using kernel module parameters.

Take a look at how i made non-contiguous allocated GART memory look contiguous to GPU, kernel-space and user-space.

The kernel module needs some IOCTLs, e.g. for setting display buffers or flip status, because it can be done ONLY with LV1 calls. I will add it later.

Host certificate, host private key and AACS LA public key are stored encrypted with AES-256-CTR in the SPU module and are decrypted when the SPU module is loaded or when it's accessed first. The AES-256-CTR key and IV are in the SPU module too.

This SPU module can create either P- or S-Block which are sent to BD drive

EID2 is passed to the SPU module

A XDR memory buffer of size 0x1000 is passed to the SPU module

4 bytes at offset 0x10 of the XDR memory buffer contain the type of block which should be produced by the SPU module

When the SPU module is finished, the XDR memory buffer contains the needed block

After the S- and P-Blocks are produced by the SPU module, they are decrypted again but with DES (CBC mode, key length is 64 bits, initialization vector length is 64 bits) before they are sent to BD drive. S$ny cuts the header and the footer of 16 bytes each from final P- and S-blocks before sending them to drive.