14.
Stepping back
14
+ Many companies have ofﬂine operations
+ Can optimize these using experiments
!
!
!
Online Experiments:
We run these all the time too.
If you are curious about on our online experimentation see Jan Overgoor’s tech talk
http://nerds.airbnb.com/tech-talks/

30.
Some takeaways
Use context to improve operations
30
+ Can investigate heterogeneity in treatment effects with higher N
+ Word of caution: can’t just compare those who were reached
by a call or email to the control (selection bias strikes again)

33.
Takeaways
+ Controlled experiments are the way to go if you want to make causal inference
+ Use them to optimize operations!
!
but:
+ Level of randomization - what impact do you want to measure?
+ Cannibalization
+ Compare the right groups - no selection bias
+ Break down results to get the most from the analysis
+ Be practical/ethical - you are dealing with real people here
33