State changed to Approved-announcement sent from Approved-announcement to be sent

2012-06-26

09

Amy Vezza

IESG has approved the document

2012-06-26

09

Amy Vezza

Closed "Approve" ballot

2012-06-26

09

Amy Vezza

Ballot approval text was generated

2012-06-26

09

Wesley Eddy

State changed to Approved-announcement to be sent from Approved-announcement to be sent::Point Raised - writeup needed

2012-06-26

09

Wesley Eddy

Ballot writeup was changed

2012-05-10

09

Amy Vezza

State changed to Approved-announcement to be sent::Point Raised - writeup needed from IESG Evaluation

2012-05-10

09

Al Morton

New version available: draft-ietf-ippm-reporting-metrics-09.txt

2012-05-10

08

Benoît Claise

[Ballot comment]- Please address the OPS-DIR review by Bert Wijnen

The one thing that concerns me a little bit is the fact that ...

[Ballot comment]- Please address the OPS-DIR review by Bert Wijnen

The one thing that concerns me a little bit is the fact that this document uses RFC2119 language. I think that is in-appropriate. Using lower case for the MUST, SHOULD and RECOMMEND in the document is perfectly fine I think.

- Support Adrian's comment regarding the title "Reporting IP Network Performance Metrics: Different Points of View"

- Next to the question "How will the results be used?", it would have been nice to ask the question "Which audience will read the results"Network Characterization = network operatorApplication Performance Estimation = application designer, service developer, etc..Actually, this is what you did, without clearly mentioning it, asking the question about "how", and answering with "two main audience categories"

-

2. Application Performance Estimation - describes the network conditions in a way that facilitates determining affects on user applications, and ultimately the users themselves. This point- of-view looks outward, toward the user(s), accepting the network as-is.

What do you mean "accepting the network as-is."?It's not because the results will be used for application performance estimation that you can't optimize your network.

- "The scope of this memo primarily covers the design and reporting of the loss and delay metrics [RFC2680] [RFC2679]."

What do you mean by design of metric? Do you mean choosing the measurement characteristics of a metric?Note: multiple occurrences of "metric design" in the draft.

o [RFC3148] includes restrictions of congestion control and the notion of unique data bits delivered, and

o [RFC5136] using a definition of raw capacity without the restrictions of data uniqueness or congestion-awareness.

It might seem at first glance that each of these metrics has an obvious audience (Raw = Network Characterization, Restricted = Application Performance), but reality is more complex and consistent with the overall topic of capacity measurement and reporting. For example, TCP is usually used in Restricted capacity measurement methods, while UDP appears in Raw capacity measurement. "

I was not sure what you meant by Raw and Restricted.

However, I saw a definition way down in the document, in section 6 and 7 Raw capacity refers to the metrics defined in [RFC5136] which do not include restrictions such as data uniqueness or flow-control response to congestion...

Restricted capacity refers to the metrics defined in [RFC3148] which include criteria of data uniqueness or flow-control response to congestion...

Please add those "definitions" in section 2.It's specifically important since RFC5136 and RFC3148 don't mention Raw/Restricted

- I learned to avoid "we", "our", "us" in RFCs.I double-checked if it's still the case with the RFC-editor. I will let you know the answer.

- I would add an extra point to "For these and other reasons, such as"Something such as:

o the ability to drill down to a specific measurement interval for deeper analysis

Justification: most of the time, when checking SLA, we check with large measurement interval, but want to ability to do a postmortem analysis

- I don't understand "Fortunately, application performance estimation activities are not adversely affected by the estimated worst-case transfer time. Although the designer's tendency might be to set the Loss Threshold at a value equivalent to a particular application's threshold, this specific threshold can be applied when post-processing the measurements. "

- "We can say that the Delay and Loss metrics are Orthogonal"Orthogonal -> orthogonal?

- section 7.4. Bulk Transfer Capacity Reporting

When BTC of a link or path is estimated through some measurement technique, the following parameters SHOULD be reported:

Also transport type, link layer type, tunneling yes/no, etc...?

- Personal preference, no need to modify the document unless you feel like it.All my customers are interested in delay, loss, and delay variation (jitter).It would have been nice to have a clear pointer in the table of content, with a clear entry "Effect of POV on the Delay Variation Metric . . . . . . . . . . . . . ." instead of addressing delay variation in "delay metric" section 5.1.3

-

Section 4.1 of [RFC3393] describes this specification and its rationale (ipdv = inter-packet delay variation in the quote below).Use IPDV (Remember you used Packet Delay Variation (PDV)) in the document, and refer to RFC5481Several ipdv instances in the draft.

-

"Network Characterization has traditionally used Poisson-distributed inter-packet spacing, as this provides an unbiased sample."

Is this correct? or Poisson-distributed start, with fixed inter-packet spacing, to match, for example, a voice/video application

2012-05-10

08

Benoît Claise

[Ballot Position Update] New position, No Objection, has been recorded for Benoit Claise

2012-05-10

08

Gonzalo Camarillo

[Ballot Position Update] New position, No Objection, has been recorded for Gonzalo Camarillo

You say "packets of type-P". Shouldn't that be "packets of type P" without the hyphen? Also, "type C"? With the hyphens, I'm not quite sure what you're talking about. Perhaps this is just unclear to someone outside the area.

2012-05-09

08

Pete Resnick

[Ballot Position Update] New position, No Objection, has been recorded for Pete Resnick

2012-05-09

08

Stephen Farrell

[Ballot comment]- 3.1: what does it mean to say the 51 seconds valuewas "calculated above" when its (now, presumably)done in 4.1 ...

[Ballot comment]- 3.1: what does it mean to say the 51 seconds valuewas "calculated above" when its (now, presumably)done in 4.1.1. (Couldn't you have arranged that 42seconds was the answer?)

- 8.2: might have been a nice thing to include somereasonable representative sample sizes for somestatistics for some measurements. Definitely toomuch to try add something with broad coverage, butone good, and one bad, set of example numbers would be a fine addition if someone had time.

2012-05-09

08

Stephen Farrell

[Ballot Position Update] New position, No Objection, has been recorded for Stephen Farrell

2012-05-09

08

Ralph Droms

[Ballot Position Update] New position, No Objection, has been recorded for Ralph Droms

2012-05-09

08

Russ Housley

[Ballot comment]

Thank you for considering the minor comments and editorial comments raised by Vijay Gurbani in the Gen-ART Review posted on 8-May-2012.

2012-05-09

08

Russ Housley

[Ballot Position Update] New position, No Objection, has been recorded for Russ Housley

2012-05-09

08

Stewart Bryant

[Ballot Position Update] New position, No Objection, has been recorded for Stewart Bryant

[Ballot Position Update] New position, No Objection, has been recorded for Robert Sparks

2012-05-08

08

Sean Turner

[Ballot Position Update] New position, No Objection, has been recorded for Sean Turner

2012-05-08

08

Ron Bonica

[Ballot Position Update] New position, No Objection, has been recorded for Ronald Bonica

2012-05-07

08

Martin Stiemerling

[Ballot Position Update] New position, Yes, has been recorded for Martin Stiemerling

2012-05-05

08

Adrian Farrel

[Ballot comment]I have no objection to the publication of this document, but I thinkit would be helpful if the document title reflected the ...

[Ballot comment]I have no objection to the publication of this document, but I thinkit would be helpful if the document title reflected the fact that the metrics being reported are IP network performance metrics.Perhaps...

Reporting IP Network Performance Metrics: Different Points of View

I also have some small Comments as follows...

---

I think the document would benefit from a further read-through to fixsome of the English and readability issues. Leaving these to the RFCEditor risks errors of meaning being introduced during the edit process.

---

Section 3.

This section gives an overview of recommendations

And...

Section 3.1.

This section gives an overview of reporting recommendations for the loss, delay, and delay variation metrics.

But...

Section 3.1

The minimal report on measurements MUST include both Loss and Delay Metrics.

This "MUST" is not a recommendation. You need to decide whether you arewriting recommendations (which seems wholy appropriate since there areno operational or interop implications of missing out some measurements)or writing requirements.

Notwithstanding the resolution of the above point, I am not convinced that you really need to use RFC 2119 language in this document.

---

Section 3.1

"We have calculated a waiting time" needs a forward reference to the place in the document where this calculation is performed.

---

Section 3.1

"99.9%-ile" is really ugly!

---

A bit puzzled by Section 4.1.1 where you have

n --- \ D = t + > (t + q ) 0 / i i --- i = 1

Presume you decided to not consider queue at the source node because youconsider it as the generator of the packets and not subject to queuing.This is slightly suspect in my opinion and depends on the nature of- the source node- the definition of the path.

Given this I wonder whether it is right to exclude q at the source orto include q at the destination. In any case, it would be helpful to explain your choices. But (of course) given the numbers being used toarrive at D using this formula including or excluding one queue time isnot really significant.

It would also be nice to note that there are n+1 nodes on your path andto clarify that q(i) is the delay due to queuing at node at the far endof the ith link.

---

Not sure why section 4.3 is present in this document. It doesn't seem toleverage or be leveraged by anything else in the document. What is more,the concluding sentence ("After waiting sufficient time, packet loss canprobably be attributed to one of these causes.") is rather vague and outof scope for the practice of measurement. Recall, the objective of ippmisto takemeasurementsandprovide reports, not to make qualatative assessments of the results.

---

Section 10

Are there no security considerations associated with running the tests over longer periods of time? What if keys roll during the measurement period? Don'tlong periods offer more chance of seeing anattack?

2012-05-05

08

Adrian Farrel

[Ballot Position Update] New position, No Objection, has been recorded for Adrian Farrel

2012-05-04

08

Brian Haberman

[Ballot Position Update] New position, No Objection, has been recorded for Brian Haberman

2012-04-28

08

Barry Leiba

[Ballot comment]Substantive suggestions; please respond to these:

-- Section 5.2 -- When both the sample mean and median are available, a comparison will ...

[Ballot comment]Substantive suggestions; please respond to these:

-- Section 5.2 -- When both the sample mean and median are available, a comparison will sometimes be informative, because these two statistics are equal only when the delay distribution is perfectly symmetrical.

I'm not a statistician, but I don't think that's true. For example, this has a symmetrical distribution with 5 as the mean and median:

1 1 4 4 5 6 6 9 9

But this also has mean and median of 5, and its distribution is not symmetrical:

1 2 3 4 5 6 6 9 9

Am I missing something?

========Editorial suggestions. No need to respond to these; take them or modify them as you please:

Throughout: there's no reason to hyphenate "point of view".

-- Introduction -- in a way that facilitates determining affects on user applications,

"effects", not "affects".

-- Section 2 -- [RFC5136] using a definition of raw capacity without the restrictions of data uniqueness or congestion-awareness.

Don't hyphenate "congestion awareness".

-- Section 3 --Don't hyphenate "long term" here. (The rule is that a compound modifier is hyphenated, but if it's not being used as a modifier (an adjective or adverb), it shouldn't be hyphenated.)

-- Section 3.1 -- We have calculated a waiting time above that should be sufficient to differentiate between packets that are truly lost or have long finite delays under general measurement circumstances, 51 seconds. Knowledge of specific conditions can help to reduce this threshold, but 51 seconds is considered to be manageable in practice.

"above"? Does this need to be re-worded? Maybe "above which it", or some such? And 51 seconds seems oddly precise: does 50 seconds really not work, and is it really not appropriate to call it 55 or 60 ? (Just asking; I have no idea of the answer here.)

For example, the 99.9%-ile minus the minimum

I suggest just spelling out "percentile" here (and in 5.2); you're not tight on column-inches. If you're worried, you can compensate by removing the extraneous "identified as" in the net paragraph.

-- Section 3.2 --

The result would ideally appear in the same form as though a continuous measurement was conducted.

Needs subjunctive mood: "had been conducted."

intervals it is possible to present the results as "metric A was less than or equal to objective X during Y% of time.

Missing closing quote.

NOTE that numerical thresholds of acceptability are not set in IETF performance work and are explicitly excluded from the IPPM charter.

Once the RFC is published, its connection with the IPPM working group is not obvious. I suggest just saying, "and are out of scope for this document," or some such.

-- Figure 2 --I suggest moving "where j is the hop number where the loop begins" out of the figure, since you already have two other "wheres" out there. You also don't say what "n" is, and should. I see from below that it's the number of hops. So make it, "where n is the total number of hops, j is the hop number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value".

-- Section 4.3 --In bullet 5, I would add a comma after "checking", to break up the length and to avoid confusion about what "and" conjoins.

-- Section 5.1.2 --

As further evidence of overlap, consider the Cumulative Distribution Function (CDF) of Delay when the value positive infinity is assigned to all lost packets.

I suggest quoting "positive infinity" to set it off clearly.

Although infinity is a familiar mathematical concept, it is somewhat disconcerting to see any time-related metric reported as infinity, in the opinion of the authors.

This is consensus of the WG, not opinion of authors, right? I suggest just ending the sentence at the comma. If you need to waffle, make it "it can be somewhat disconcerting".

-- Section 5.3 -- the most efficient practice is to distinguish between truly lost and delayed packets with a sufficiently long waiting time, and to designate the delay of non-arriving packets as undefined.

Again, it's easy to misread what the "and" conjoins. How about this way?:NEW the most efficient practice is to distinguish between packets that are truly lost and those that are delayed with a sufficiently long waiting time, and to designate the delay of non-arriving packets as undefined.

-- Section 7.5 --Last paragraph begins with a lower-case "w".

2012-04-28

08

Barry Leiba

Ballot comment text updated for Barry Leiba

2012-04-28

08

Barry Leiba

[Ballot comment]Substantive suggestions; please respond to these:

-- Section 5.2 -- When both the sample mean and median are available, a comparison will ...

[Ballot comment]Substantive suggestions; please respond to these:

-- Section 5.2 -- When both the sample mean and median are available, a comparison will sometimes be informative, because these two statistics are equal only when the delay distribution is perfectly symmetrical.

I'm not a statistician, but I don't think that's true. For example, this has a symmetrical distribution with 5 as the mean and median:1 1 4 4 5 6 6 9 9But this also has mean and median of 5, and its distribution is not symmetrical:1 2 3 4 5 6 6 9 9Am I missing something?

========Editorial suggestions. No need to respond to these; take them or modify them as you please:

Throughout: there's no reason to hyphenate "point of view".

-- Introduction -- in a way that facilitates determining affects on user applications,

"effects", not "affects".

-- Section 2 -- [RFC5136] using a definition of raw capacity without the restrictions of data uniqueness or congestion-awareness.

Don't hyphenate "congestion awareness".

-- Section 3 --Don't hyphenate "long term" here. (The rule is that a compound modifier is hyphenated, but if it's not being used as a modifier (an adjective or adverb), it shouldn't be hyphenated.)

-- Section 3.1 -- We have calculated a waiting time above that should be sufficient to differentiate between packets that are truly lost or have long finite delays under general measurement circumstances, 51 seconds. Knowledge of specific conditions can help to reduce this threshold, but 51 seconds is considered to be manageable in practice.

"above"? Does this need to be re-worded? Maybe "above which it", or some such? And 51 seconds seems oddly precise: does 50 seconds really not work, and is it really not appropriate to call it 55 or 60 ? (Just asking; I have no idea of the answer here.)

For example, the 99.9%-ile minus the minimum

I suggest just spelling out "percentile" here (and in 5.2); you're not tight on column-inches. If you're worried, you can compensate by removing the extraneous "identified as" in the net paragraph.

-- Section 3.2 --

The result would ideally appear in the same form as though a continuous measurement was conducted.

Needs subjunctive mood: "had been conducted."

intervals it is possible to present the results as "metric A was less than or equal to objective X during Y% of time.

Missing closing quote.

NOTE that numerical thresholds of acceptability are not set in IETF performance work and are explicitly excluded from the IPPM charter.

Once the RFC is published, its connection with the IPPM working group is not obvious. I suggest just saying, "and are out of scope for this document," or some such.

-- Figure 2 --I suggest moving "where j is the hop number where the loop begins" out of the figure, since you already have two other "wheres" out there. You also don't say what "n" is, and should. I see from below that it's the number of hops. So make it, "where n is the total number of hops, j is the hop number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value".

-- Section 4.3 --In bullet 5, I would add a comma after "checking", to break up the length and to avoid confusion about what "and" conjoins.

-- Section 5.1.2 --

As further evidence of overlap, consider the Cumulative Distribution Function (CDF) of Delay when the value positive infinity is assigned to all lost packets.

I suggest quoting "positive infinity" to set it off clearly.

Although infinity is a familiar mathematical concept, it is somewhat disconcerting to see any time-related metric reported as infinity, in the opinion of the authors.

This is consensus of the WG, not opinion of authors, right? I suggest just ending the sentence at the comma. If you need to waffle, make it "it can be somewhat disconcerting".

-- Section 5.3 -- the most efficient practice is to distinguish between truly lost and delayed packets with a sufficiently long waiting time, and to designate the delay of non-arriving packets as undefined.

Again, it's easy to misread what the "and" conjoins. How about this way?:NEW the most efficient practice is to distinguish between packets that are truly lost and those that are delayed with a sufficiently long waiting time, and to designate the delay of non-arriving packets as undefined.

-- Section 7.5 --Last paragraph begins with a lower-case "w".

2012-04-28

08

Barry Leiba

[Ballot Position Update] New position, No Objection, has been recorded for Barry Leiba

(1.a) Who is the Document Shepherd for this document? Has the Document Shepherd personally reviewed this version of the document and, in particular, does he or she believe this version is ready for forwarding to the IESG for publication?

Document sheperd is Henk Uijterwaal.

(1.b) Has the document had adequate review both from key WG members and from key non-WG members?

Yes, the document was reviewed by some members of the working group.

(1.c) Does the Document Shepherd have concerns that the document needs more review from a particular or broader perspective,

No.

(1.d) Does the Document Shepherd have any specific concerns or issues with this document that the Responsible Area Director and/or the IESG should be aware of?

No.

(1.e) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Good, the topic has been under discussion for years and the consensusis that this document (and the related one on short term reports) reflect current thinking about this topic.

(1.g) Has the Document Shepherd personally verified that the document satisfies all ID nits?

Yes, the pre RFC 5378 disclaimer is, afaik, necessary.

(1.h) Has the document split its references into normative and informative?

Yes Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state?

No. I don't see an issue with the informative reference to draft-ietf-ippm-reporting.

(1.i) Has the Document Shepherd verified that the document IANA consideration section exists and is consistent with the body of the document?

Yes, it is empty.

(1.j) Has the Document Shepherd verified that sections of the document that are written in a formal language, such as XML code, BNF rules, MIB definitions, etc., validate correctly in an automated checker?

N/A

(1.k) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up? Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary

Consumers of IP network performance metrics have many different uses in mind. The memo provides "long-term" reporting considerations (e.g, days, weeks or months, as opposed to 10 seconds), based on analysis of the two key audience points-of-view. It describes how the audience categories affect the selection of metric parameters and options when seeking info that serves their needs.

Working Group SummaryThe normal WG process was followed and the document has been discussed forseveral years. The document as it is now, reflects WG consensus, with nothingspecial worth noticing.