Menu

An article form June 2014 on how Optimizely and others only do one-tail tests, which simplifies the conclusion.

Entertaining article, here are some take-aways:

Do a test that is two-tailed (to see if there is significance for the opposite as well)

“The short answer is that with a two-tailed test, you are testing for the possibility of an effect in two directions, both the positive and the negative. One-tailed tests, meanwhile, allow for the possibility of an effect in only one direction, while not accounting for an impact in the opposite direction.”

Some of the the hacker news comments (from January 2016) are interesting too, one mentioning that showing the variation is essential:

“In my view, the issue is not one-tail vs two-tail tests, or sequential vs one-look tests at all. The issue is a failure to quantify uncertainty.

Optimizely (last time I looked), our old reports, and most other tools, all give you improvement as a single number. Unfortunately that’s BS. It’s simply a lie to say “Variation is 18% better than Control” unless you had facebook levels oftraffic. An honest statement will quantify the uncertainty: “Variation is between -4.5% and +36.4% better than Control”.

(..)“We just report credible intervals. We find that to be the only honest choice.

Another point to raise is the issue on the impact on the Long Time Value of the user too. So, yes we got a higher CTR in the test, but how is the impact on LTV, is it the right kind of users who are clicking on that button?

Mobile App Tracking, a company that boasts clientele like Zynga and LivingSocial, has proposed device fingerprinting. This approach is rather scatterbrained, as it needs apps to collect various bits of data about your iPhone (location, IP address, iOS version, etc.) and combine them together into one unique identifier. Unfortunately, users cannot easily opt out of fingerprinting or reset their device fingerprint altogether. There’s plenty of room for discrepancies since all of these bits of data can be meshed together and lose their accuracy over time. Device fingerprinting also requires licensing fees, so don’t expect it to gain widespread adoption.

And the history continues, idFA (Apple’s ID For Advertising) , articel from August 2013

The main drawback to dynamic device IDs is the inability to support cross-device and cross-channel advertising performance attribution.

There are a few industry players working to address the attribution problem by creating an advertiser-agnostic central clearinghouse, where consumers opt in to a universal ID that can be used for targeted advertising and to measure engagement. This approach suffers from a heavy reliance on consumer and market education, and perhaps unrealistic expectations for consumer adoption and industry standardization.

Until then, we have to go back to our roots in science and math and to refocus on users rather than devices to overcome the limitations of cookies in mobile. Applying statistical probabilities to user behaviors online enables advertisers to estimate the likelihood that a user of one device is the same user of any other device, including tablets, smartphones, digital TV, desktop or any other digital device.

All in a day’s work. Imagine building a house in just five hours, and I’m not talking about a Lego mansion, I’m talking about a real structure that you can live in. Dutch designer Pieter Stoutjesdijk has created an emergency shelter that can be snapped together in just five hours, reports Fast Company. Stoutjesdijk’s open source design is meant to bring shelter to people who lose their homes in a disaster. And while we’re on the topic of open source construction, check out this interview with WikiHouse co-founder Alastair Parvin, who shares details on where the open source construction set is heading.

If we don’t want a total surveillance society, we must consider surveillance a kind of social pollution, and limit the surveillance impact of each new digital system just as we limit the environmental impact of physical construction.

For example: “Smart” meters for electricity are touted for sending the power company moment-by-moment data about each customer’s electric usage, including how usage compares with users in general. This is implemented based on general surveillance, but does not require any surveillance. It would be easy for the power company to calculate the average usage in a residential neighborhood by dividing the total usage by the number of subscribers, and send that to the meters. Each customer’s meter could compare her usage, over any desired period of time, with the average usage pattern for that period. The same benefit, with no surveillance!

First point/theory, as mentioned by Paul Rayment, the user scenario when watching TV is that the smartphone or tablet is out too, it is a dual screen TV watching world out there, so the TV ad is darn effective (could be interesting to see download numbers while the ad is aired and immediately after).

Second point/theory, playing a casual game is filled with stigma in the public sphere (your article is a great example of this; you are “embarrassed” to tell what level you are on!), putting casual games ads on TV makes playing the game less stigmatized, it is giving it ‘credibility’, thus making people play more and even give them an opportunity to dare spread it through word of mouth.

Thanks for a good article, would you know whether this was the first time in the whole of US (“last week”) that the commercial aired on TV?