The SLA (Service Level Agreement) topic is already well covered – books have been written and libraries filled on this subject- so I will try to avoid writing a book of my own. (BTW my favorite reference is: “Foundations of Service Level Management” by Rick Sturm and Wayne Morris )

In my experience SLAs are very important, and (if not abused) they can be a great tool to align Clients and Suppliers around FACTS and DATA.

While VP of QoS at DoubleClick I was responsible for reviewing, approving and monitoring our external and internal SLAs; we had over 400 customers with SLAs, and each had at least 2 or 3 objectives!

One of my first tasks while setting up the QoS group was finding an effective way to measure the performance of third party providers – and employing that technique in SLAs.

The challenge was that clients would look at their site performance and notice spikes and they would attribute it to our system, meanwhile our performance chart would not show any problems. We couldn’t correlate the two charts, and therefore we couldn’t come to an agreement when it was our problem, and when it was someone else’s problem.

Working with an amazing statistician, Matt Briggs we created a methodology we called Differential Performance Measurement. (DPM)

The philosophy behind DPM was to be able to measure, as accurately as possible, the performance and availability of Doubleclick’s services and their impact on the pages of our customers, making sure we were responsible and accountable for the things we had control over and there was no finger pointing.

The methodology added context to the measurements. DPM introduced clarity and comparison, removing absolute performance numbers from the SLAs.

Recipe for Differential Performance Measurement (Example with an Advert):

1- Take two pages, one without ads and one with one ad call.

Page A = without ad

Page B = with ad

2- Make sure the pages do not contain any other third-party references (CDNs etc.)

3- Make sure the page sizes (in KB) are the same

4- “Bake” – Measure response times for both pages and you get the following metrics:

Differential Response (DR) will be (Response Time of page B) minus (Response Time of page A)

Post navigation

You Might Also Like

Comments (6)

On our site we are using DoubleClick to serve graphical ads. All our attempts to use Keynote to measure ads SLA were not very productive because we realized that DC is requesting ads that call other ads and also send beacons. One Dart Tag call might triger 4-6 other java script calls. Keynote gives us data on each of the request but it’s impossible to group them to know which request came from what Dart Tag call.
Any suggestions on how to deal with this?

DPM looks like it gives you insight into 3rd party content latency spikes by monitoring changes in the differential (spikes in the red line).

Ad latency is one factor affecting overall site performance, but using certain techniques (e.g. the “friendly iframe”), one can effectively disassociate ad latency from the base page load, which indicates that ad-loading implementation has a bearing on overall site performance. However, ads also fight for scarce resources on the client side, including connections and the single javascript thread.

While we can measure and attribute performance deviation using differential measuring techniques, what remedies are there to improve the baseline itself?

We have a page that currently loads 6 ads. DPM will help us better understand when the differential changes, but we’re also looking for ways to simply improve the regular differential. I’m pretty sure that there are some optimizations we can do to make improvements, but given various constraints, 6 ads may just simply be “heavy”. These 6 ads cumulatively load about 710 lines of JS. I’m wondering how much of an impact this is having on our baseline.

Does anyone out there have any experience with the sides of the equation that aren’t strictly latency oriented, assuming that we can utilize “friendly iframes” to decouple ad latency from blocking the main page content’s onload event?