Thursday, August 8, 2013

This post is about one of the last thing that were missing in my own Squeak/Pharo COG Virtual Machine hack which use native order 32 bits digits Large Integers: that is image segment loading/saving support.On little endian machines, this is not a problem because native 32 bits word order matches Large Integers byte order. So loading image segment on little endian machines just work when the segment has been saved from a little endian machine, or from an old VM without native order 32 bits digits Large Integers support.This is only a problem on big endian machines.The only hurdle was to decide how to mark the segment so that it can be loaded correctly.But let's explain a bit what's going on with byte order at load time...

Swapping byte order at image segment load time:

Loading an image segment from a different endianness requires byte-swapping of byte objects. This is because the primitive for loading image segment first swap every 32 bits word in the segment (by invoking ObjectMemory>>reverseBytesFrom:to:) as if every object were 32 bits word-array or composed exclusively of 32 bits object oriented pointers. But this is not the case for byte-arrays, and must be undone for those objects.This is further processed by invoking methodObjectMemory>>byteSwapByteObjectsFrom:to:flipFloatsIf:The last parameter of this method is for swapping the two words composing a Float (64 bits IEEE 754 double precision format) if ever the segment comes from an old image. Indeed, in old images, Float were always stored in Big Endian format, whatever VM endianness...

This byte swapping is triggered by a test of image segment version which is stored into first 32 bits word data, versus current ObjectMemory imageSegmentVersion.Let's see this method:imageSegmentVersion | wholeWord | "a more complex version that tells both the word reversal and the endianness of the machine it came from. Low half of word is 6502. Top byte is top byte of #doesNotUnderstand: on this machine. ($d on the Mac or $s on the PC)"

Hacking the image segment version

Since we are in COG, we will hack NewObjectMemory>>imageSegmentVersion. We will use one of the high bits of unused byte (the second most significant), the 7th bit starting at 1, (weight 64), that is the 22nd one in the whole word.Note that we already used the 7th bit of image header flags (the 7th 32 bits word of the image), so our VM hack is somehow self consistent.imageSegmentVersion | wholeWord flagLargeIntsAreStoredInBigEndian | "a more complex version that tells both the word reversal and the endianness of the machine it came from. Low half of word is 6502. Top byte is top byte of #doesNotUnderstand: on this machine. ($d on the Mac or $s on the PC)"

We could have factored this loop with Float swapping, but this will be an optimization for a later time (though this would have saved a stupid copy paste bug in version 319).

Deciding when to swap the bytes

Now we have three possible format for Large Integers:

little endian for segments saved from little endian VM (16r7300YYYY)

little endian for segments saved from old big endian VM (16r6400YYYY)

big endian for segments saved from new big endian VM (16r6440YYYY)

We want to swap large integer bytes in those cases:

VM is big endian, and segment is big endian with old little endian large integers

VM is big endian and segment is little endian

VM is little endian and segment is big endian with new big endian large integers

In all cases, this is when 22nd bit of image segment data does not match that of current NewObjectMemory imageSegmentVersion. So we modify these lines of NewObjectMemory>>loadImageSegmentFrom:outPointers:handling the byte reversal:

"Reverse the Byte type objects if the data is from opposite endian machine. Revese the words in Floats if from an earlier version with different Float order. Test top byte. $d on the Mac or $s on the PC. Rest of word is equal."swapLargeInts := (self vmEndianness << 22) ~= (data bitAnd: 1 << 22). (data >> 24) = (self imageSegmentVersion >> 24) ifTrue: "Need to swap floats if the segment is being loaded into a little-endian VM from a version that keeps Floats in big-endian word order as was the case prior to the 6505 image format." [(self isPlatformFloatOrderVersion: (data bitAnd: 16rFFFF "low 2 bytes")) ifFalse: [self vmEndianness ~= 1 "~= 1 => little-endian" ifTrue: [segOop := self oopFromChunk: segmentWordArray + BaseHeaderSize + BytesPerWord. self wordSwapFloatsFrom: segOop to: endSeg + BytesPerWord]]."Need to swap large integers if both segment and vm are big endian, but segment did not use native 32 bit word order for large integers" swapLargeInts ifTrue: [ self wordSwapLargeIntsFrom: segOop to: endSeg + BytesPerWord]] ifFalse: "Reverse the byte-type objects once" ["Change of endianness: need to swap large integers, except if segment is big endian but did not use 32 bit word order for large integers" segOop := self oopFromChunk: segmentWordArray + BaseHeaderSize + BytesPerWord. "Oop of first embedded object" self byteSwapByteObjectsFrom: segOop to: endSeg + BytesPerWordbutLargeIntIf: swapLargeInts "don't unswap already swapped large integers" flipFloatsIf: (self isPlatformFloatOrderVersion: (data bitAnd: 16rFFFF "low 2 bytes"))].Last precision, when we want to effectively swap large integers, then we just omit to unswap them, because the save/load mechanism already swapped them.Code for COG can be found at http://smalltalkhub.com/#!/~nice/NiceVMExperiments/versions/VMMaker.oscog-nice.320.Similar code for interpreter VM is at http://smalltalkhub.com/#!/~nice/NiceVMExperiments/versions/VMMaker-nice.324.Lat thing, this code is currently untested on big endian machines: they are becoming quite rare nowadays and I don't own any.