Sunday, 29 April 2018

Last year we made rr use "CPUID faulting" to record and replay the results of CPUID instructions. This has enabled a reasonable level of rr trace portability in practice. However, we've run into some interesting limitations, some of which we've addressed and others that I don't know how to address.

Sometimes the recording CPUs support features that are missing on the CPUs you want to eventually replay on. If your application detects and uses those features, replay won't work. Fortunately, because CPUID faulting lets us control the results of application CPUID invocations, it was easy to extend rr with the ability to mask off CPUID feature bits during recording. I added recording options --disable-cpuid-features, --disable-cpuid-features-ext, and --disable-cpuid-features-xsave to do this. To make it easier to determine which bits to mask off, I also added a new command rr cpufeatures which you can run on a replay machine to print the command line options you should use for recording, to disable features unsupported on the replay machine. This seems to work reasonably well.

Unfortunately there's a portability hazard related to the XSAVE instruction. The size of the memory range written by XSAVE depends on the XSAVE features supported by the CPU and enabled by the OS via the XCR0 register. This can be different on the recording and replay machines and there's nothing rr can do about it! User-space code can (and should!) use CPUID queries to determine what that size is, but overriding what we report doesn't change what the CPU will actually write. Even though applications shouldn't really depend on the what's written by XSAVE, as long as XSAVE/XRSTOR pairs actually restore state as expected, in practice the size of XSAVE writes affects uninitialized values on the stack which leak into registers which cause rr to detect divergence and abort.

A possible work-around is to mask off XSAVE feature bit in CPUID results, so well-behaved user-space programs don't use XSAVE at all. Unfortunately that also effectively disables AVX and other XSAVE-dependent features :-(.

A possible fix is to add a Linux kernel feature to let us set the XCR0 value for specific processes, i.e. disable XSAVE state saving for some XSAVE components. Then during recording we could limit the XSAVE components to those that are supported by the replay machine. Unfortunately, from experience we know that adding new a kernel API upstream is quite difficult, especially those that require adding code to context switches.

A simpler kernel patch would be to provide a boot command-line option to mask bits out of XCR0 for all processes.

Another possible work-around would be to record in a virtual machine whose XCR0 is specifically configured to match the replay machine's. I haven't tried it but I assume it's possible; VM migration would require this.

For now I plan to just have rr print an error during replay when XSAVE is enabled and the replay machine's XCR0 features are not equal to the recording machine's.

Update I realized that the XSAVEC instruction ("XSAVE compressed") avoids the above issues. State components that are not in use do not affect what is written by XSAVEC. If a state component is actually in use during recording but not supported during replay, replay is already impossible. Therefore applications that stick to XSAVEC and avoid XSAVE will not incur the above problems. The only user-level use of XSAVE I currently know of is in the glibc dynamic loader, which prefers XSAVEC if available; therefore recording (and replaying) on machines supporting XSAVEC should generally work fine regardless of what XSAVE features are enabled via XCR0. The situation is not as bad as I feared.

Saturday, 21 April 2018

Last week I did the Heaphy Track again with some friends and family. We did the track west to east this time, staying overnight in Westport on Saturday and spending the following nights on the trail in Heaphy Hut, Mckay Hut, and Perry Saddle Hut. The weather forecast was pretty bad from outset, predicting heavy rain and high winds for Monday, Tuesday and Wednesday. In the end the weather wasn't as bad as forecast, but it did rain a fair bit on those days and we had a major thunderstorm on Sunday night — glad to be in the hut! For some of our group it was their first multi-day tramp, and covering 80km in four days in those conditions wasn't the easiest introduction, but I'm confident everyone had at least a memorable time, and probably a good one :-). We had a bit of everything: beaches, forest, rivers, giant carnivorous snails, lots of wekas, limestone caves (next to Gouland Downs Hut), wetas, rain, sun, views of snow-capped hills. In huts we played lots of Bang, some Citadels and San Juan, talked to interesting people from many countries, and cooked some pretty good food.

Tuesday, 10 April 2018

Today I discovered that an Australian company called Payment Express has started offering, in addition to credit-card payment processing, a feature called "Account2Account". With this feature, customers enter their online banking credentials into Payment Express' Web site which then performs a payment transaction on the customer's behalf. This is insane and I don't know why banks allow it.

The security FAQ presented to customers (which I can't find a public URL for) emphasizes that Payment Express does not store the customer's credentials or other information. Good, but the problem is that even if customers can completely trust Payment Express (and I don't know why they should; Payment Express' terms and conditions disclaim all liability), any workflow that trains customers to enter their banking credentials into a Web site other than their bank's site makes them vulnerable to phishing attacks. Online banking phishing attacks are ubiquitous. Even worse, Payment Express advertises "Payment page look and feel customisable via an online wizard", which suggests that the appearance of pages presented to customers can vary, making those customers even more vulnerable to phishing. Payment Express doesn't even use an EV certificate.

Maybe banks allow this because they get a bigger cut of the transactions than they do with a credit-card transaction, enough to cover the cost to them of any increase in phishing losses? If so I don't know how they're calculating the cost to our entire social ecosystem of people being trained to enter critical login credentials into random Web sites.