on the demand side of the advertising
market (that yields revenue to advertising platforms), the work on the analysis
of user behavior is significantly more
sparse. Here, we briefly discuss some
recent empirical results on the analysis
of user behavior. This aspect of the market leaves a plethora of opportunities
for future research.

The basic idea that explains the
emergence of this problem is that user
actions (such as the clicking on an
ad) are correlated with observable and
unobservable user characteristics. At
the same time, ad delivery ensures ads
are, by their very design, delivered to customers or during time periods that have
different purchasing probabilities as
compared to the baseline population. As
a result, the straightforward approach
based on predicting user action by using
user characteristics can lead to overestimating the impact of advertising as
shown in Blake et al. 7 and Lewis et al. 35

To validate observational estimates
of user behavior (such as the clickthrough rates for different advertising
slots), advertising platforms rely on
fully randomized experiments. These
experiments are used by advertising
platforms to infer important quantities
of interest. For example, search engines
exogenously re-order ad ranking and
limit the number of ads shown in order
to produce training data for many algorithms that govern ad delivery systems.
These experiments are conducted on a
small percentage of traffic so as to limit
revenue risk, but nonetheless, still capture millions of searches. The experiments provide ground truth estimates of
user behavior such as the causal effect of
an ad as well as the impact of “organic”
results on ad effectiveness. However,
given their size, these estimates often
need to be aggregated over many advertisers to obtain reasonable precision.

New directions for further work.Recently, advertising platforms havebegun shifting toward advertiser goal-based pricing. In other words, if anadvertiser’s ultimate goal is sales tocustomers, then an advertiser maynot be interested in the clicks per se.However, the noise in advertiser-spe-cific estimates makes “reasonable size”experiments inadequate for support-ing market design solutions that wouldhave advertiser only pay for marginalclicks (i.e., the clicks that were actuallyfunction of the rationalizable set bydefining the following functions thatcan be computed directly from the auc-tion data via simulationThese functions characterize how anaverage outcome in T auctions changeswhen bidder i switches to a fixed alter-nate bid b¢ from an actually applied bidsequence. The characterization of therationalizable set is given in the followingtheorem.

Theorem 2. Under monotonicity and continuity of DP(×) and DC(×) the support function of NR with u=(u1, u2)T and u= 1 is

.

This theorem establishes that valuations and algorithm parameters for
ε-regret algorithms can be recovered
from the data (by computing functions
DC(×) and DP(×) ). If the bids constitute
the Nash equilibrium, we can pinpoint
the bidders’ value per click. As explained
earlier, the initial stage of learning may
not be a Nash equilibrium and there
will be an entire range of values compatible with the data. At the same time,
the characterization of this range of
values and the entire rationalizable
set for learning bidders are reduced
to evaluation of two one-dimensional
functions from the data. We can use
efficient numerical approximation for
such an evaluation. The corresponding
error in the estimated rationalizable
set will be determined by the estimation error of functions DC(×) and DP(×).

Platform Design and User Behavior
Inference of user actions. As discussed,
the clickthrough rate measuring the
probability that a given user will click
on a given ad is a crucial input in pricing and allocation rules for advertising auctions. Advertising platforms
use sophisticated machine learning
tools to make such predictions on a
user-by-user basis. However, despite a
significant amount of effort and sophisticated statistical approaches, measuring the casual impact of advertising on
the actions of users has been shown to
be exceedingly difficult. 25 In addition,
since a larger emphasis has been placed

Despite a significantamount of effortand sophisticatedstatisticalapproaches,measuring thecausal impact ofadvertising on theactions of usershas been shownto be exceedinglydifficult.