idnits 2.16.02
/tmp/draft-ietf-ipfix-reducing-redundancy-04.txt:
Checking boilerplate required by RFC 5378 and the IETF Trust (see
https://trustee.ietf.org/license-info):
----------------------------------------------------------------------------
** It looks like you're using RFC 3978 boilerplate. You should update this
to the boilerplate described in the IETF Trust License Policy document
(see https://trustee.ietf.org/license-info), which is required now.
-- Found old boilerplate from RFC 3978, Section 5.1 on line 19.
-- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on
line 1392.
-- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1403.
-- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1410.
-- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1416.
Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/checklist :
----------------------------------------------------------------------------
No issues found here.
Miscellaneous warnings:
----------------------------------------------------------------------------
== The copyright year in the IETF Trust Copyright Line does not match the
current year
== Line 297 has weird spacing: '...rmation model...'
-- The exact meaning of the all-uppercase expression 'NOT REQUIRED' is not
defined in RFC 2119. If it is intended as a requirements expression, it
should be rewritten using one of the combinations defined in RFC 2119;
otherwise it should not be all-uppercase.
== Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD',
or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please
use uppercase 'NOT' together with RFC 2119 keywords (if that is what you
mean).
Found 'MUST not' in this paragraph:
If UDP is selected as the transport protocol, the
commonPropertiesID Template Withdraw Messages MUST not be used, as this
method is inefficient due to the unreliable nature of UDP.
-- The document seems to lack a disclaimer for pre-RFC5378 work, but may
have content which was first submitted before 10 November 2008. If you
have contacted all the original authors and they are all willing to grant
the BCP78 rights to the IETF Trust, then this is fine, and you can ignore
this comment. If not, you may need to add the pre-RFC5378 disclaimer.
(See the Legal Provisions document at
https://trustee.ietf.org/license-info for more information.)
-- The document date (May 21, 2007) is 4387 days in the past. Is this
intentional?
Checking references for intended status: Informational
----------------------------------------------------------------------------
== Outdated reference: draft-ietf-ipfix-protocol has been published as RFC
5101
== Outdated reference: draft-ietf-ipfix-info has been published as RFC 5102
== Outdated reference: draft-ietf-psamp-protocol has been published as RFC
5476
== Outdated reference: draft-ietf-ipfix-as has been published as RFC 5472
== Outdated reference: draft-ietf-ipfix-architecture has been published as
RFC 5470
== Outdated reference: draft-ietf-psamp-info has been published as RFC 5477
== Outdated reference: draft-ietf-psamp-sample-tech has been published as
RFC 5475
== Outdated reference: draft-ietf-psamp-framework has been published as RFC
5474
-- Obsolete informational reference (is this intentional?): RFC 2960
(Obsoleted by RFC 4960)
Summary: 1 error (**), 0 flaws (~~), 11 warnings (==), 9 comments (--).
Run idnits with the --verbose option for more detailed information about
the items above.
--------------------------------------------------------------------------------
2 IPFIX Working Group E. Boschi
3 Internet-Draft Hitachi Europe
4 Intended status: Informational L. Mark
5 Expires: November 22, 2007 Fraunhofer FOKUS
6 B. Claise
7 Cisco Systems, Inc.
8 May 21, 2007
10 Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet
11 Sampling (PSAMP) Reports
12 draft-ietf-ipfix-reducing-redundancy-04.txt
14 Status of this Memo
16 By submitting this Internet-Draft, each author represents that any
17 applicable patent or other IPR claims of which he or she is aware
18 have been or will be disclosed, and any of which he or she becomes
19 aware will be disclosed, in accordance with Section 6 of BCP 79.
21 Internet-Drafts are working documents of the Internet Engineering
22 Task Force (IETF), its areas, and its working groups. Note that
23 other groups may also distribute working documents as Internet-
24 Drafts.
26 Internet-Drafts are draft documents valid for a maximum of six months
27 and may be updated, replaced, or obsoleted by other documents at any
28 time. It is inappropriate to use Internet-Drafts as reference
29 material or to cite them other than as "work in progress."
31 The list of current Internet-Drafts can be accessed at
32 http://www.ietf.org/ietf/1id-abstracts.txt.
34 The list of Internet-Draft Shadow Directories can be accessed at
35 http://www.ietf.org/shadow.html.
37 This Internet-Draft will expire on November 22, 2007.
39 Copyright Notice
41 Copyright (C) The IETF Trust (2007).
43 Abstract
45 This document describes a bandwidth saving method for exporting flow
46 or packet information using the IP Flow Information Export (IPFIX)
47 protocol. As the Packet Sampling (PSAMP) protocol is based on IPFIX,
48 these considerations are valid for PSAMP exports as well.
50 This method works by separating information common to several flow
51 records from information specific to an individual flow record.
52 Common flow information is exported only once in a data record
53 defined by an option template, while the rest of the specific flow
54 information is associated with the common information via a unique
55 identifier.
57 Table of Contents
59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
60 1.1. IPFIX Documents Overview . . . . . . . . . . . . . . . . . 3
61 1.2. PSAMP Documents Overview . . . . . . . . . . . . . . . . . 4
62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
63 2.1. Terminology Summary Table . . . . . . . . . . . . . . . . 8
64 2.2. IPFIX Flows versus PSAMP Packets . . . . . . . . . . . . . 9
65 3. Specifications for bandwidth saving information export . . . . 9
66 3.1. Problem Statement and High Level Solution . . . . . . . . 9
67 3.2. Data Reduction technique . . . . . . . . . . . . . . . . . 11
68 4. Transport Protocol Choice . . . . . . . . . . . . . . . . . . 12
69 4.1. PR-SCTP . . . . . . . . . . . . . . . . . . . . . . . . . 12
70 4.2. UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
71 4.3. TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
72 5. commonPropertiesID Management . . . . . . . . . . . . . . . . 13
73 6. The Collecting Process Side . . . . . . . . . . . . . . . . . 14
74 6.1. UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
75 6.2. TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
76 7. Advanced Techniques . . . . . . . . . . . . . . . . . . . . . 16
77 7.1. Multiple Data Reduction . . . . . . . . . . . . . . . . . 16
78 7.2. Cascading Common Properties . . . . . . . . . . . . . . . 19
79 8. Export and Evaluation Considerations . . . . . . . . . . . . . 19
80 8.1. Transport Protocol Choice . . . . . . . . . . . . . . . . 20
81 8.2. Reduced Size Encoding . . . . . . . . . . . . . . . . . . 20
82 8.3. Efficiency Gain . . . . . . . . . . . . . . . . . . . . . 20
83 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
84 10. Security Considerations . . . . . . . . . . . . . . . . . . . 21
85 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 21
86 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
87 12.1. Normative References . . . . . . . . . . . . . . . . . . . 22
88 12.2. Informative References . . . . . . . . . . . . . . . . . . 22
89 Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 23
90 A.1. Per Flow Data Reduction . . . . . . . . . . . . . . . . . 23
91 A.2. Per Packet Data Reduction . . . . . . . . . . . . . . . . 27
92 A.3. commonPropertiesID Template Withdrawal Message . . . . . . 30
93 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 31
94 Intellectual Property and Copyright Statements . . . . . . . . . . 32
96 1. Introduction
98 The IPFIX working group has specified a protocol to export IP Flow
99 information [I-D.ietf-ipfix-protocol]. This protocol is designed to
100 export information about IP traffic flows and related measurement
101 data, where a flow is defined by a set of key attributes (e.g. source
102 and destination IP address, source and destination port, etc.).
103 However, thanks to its template mechanism, the IPFIX protocol can
104 export any type of information, as long as the information element is
105 specified in the IPFIX Information Model [I-D.ietf-ipfix-protocol] or
106 registered with IANA.
108 Regardless of the fields content, flow records with common properties
109 export the same fields in every single data record. These common
110 properties may represent values common to a collection of flows or
111 packets, or values that are invariant over time. Note that the
112 common properties don't represent the list of flow keys, which are
113 used to define a flow definition: however, the common properties may
114 contain some of the flow keys. The reduction of redundant data from
115 the export stream can result in a significant reduction of the
116 transferred data.
118 This draft specifies a way to export these invariant or common
119 properties only once, while the rest of the flow specific properties
120 are exported in regular data records. Unique common properties
121 identifiers are used to link data records and the common attributes.
123 The proposed method is applicable to IPFIX flow and to PSAMP per
124 packet information, without any changes to both the IPFIX and PSAMP
125 protocol specifications.
127 1.1. IPFIX Documents Overview
129 The IPFIX Protocol [I-D.ietf-ipfix-protocol] provides network
130 administrators with access to IP flow information. The architecture
131 for the export of measured IP flow information out of an IPFIX
132 exporting process to a collecting process is defined in the IPFIX
133 Architecture [I-D.ietf-ipfix-architecture], per the requirements
134 defined in RFC 3917 [RFC3917]. The IPFIX Architecture
135 [I-D.ietf-ipfix-architecture] specifies how IPFIX data record and
136 templates are carried via a congestion-aware transport protocol from
137 IPFIX exporting processes to IPFIX collecting process. IPFIX has a
138 formal description of IPFIX information elements, their name, type
139 and additional semantic information, as specified in the IPFIX
140 Information Model [I-D.ietf-ipfix-info]. Finally the IPFIX
141 Applicability Statement [I-D.ietf-ipfix-as] describes what type of
142 applications can use the IPFIX protocol and how they can use the
143 information provided. It furthermore shows how the IPFIX framework
144 relates to other architectures and frameworks.
146 1.2. PSAMP Documents Overview
148 The document "A Framework for Packet Selection and Reporting"
149 [I-D.ietf-psamp-framework], describes the PSAMP framework for network
150 elements to select subsets of packets by statistical and other
151 methods, and to export a stream of reports on the selected packets to
152 a collector. The set of packet selection techniques (sampling,
153 filtering, and hashing) supported by PSAMP are described in "Sampling
154 and Filtering Techniques for IP Packet Selection"
155 [I-D.ietf-psamp-sample-tech]. The PSAMP protocol
156 [I-D.ietf-psamp-protocol] specifies the export of packet information
157 from a PSAMP exporting process to a PSAMP collecting process. Like
158 IPFIX, PSAMP has a formal description of its information elements,
159 their name, type and additional semantic information. The PSAMP
160 information model is defined in [I-D.ietf-psamp-info]. Finally
161 [I-D.ietf-psamp-mib] describes the PSAMP Management Information Base.
163 2. Terminology
165 The terms in this section are in line with the IPFIX terminology
166 section in the IPFIX [I-D.ietf-ipfix-protocol], and PSAMP
167 [I-D.ietf-psamp-protocol] protocol specifications. Note that this
168 document selected the IPFIX definition of the term Exporting Process
169 [I-D.ietf-ipfix-protocol], as this definition is more generic than
170 the PSAMP definition [I-D.ietf-psamp-protocol].
172 Observation Point: An Observation Point is a location in the
173 network where IP packets can be observed. Examples include: a
174 line to which a probe is attached, a shared medium, such as an
175 Ethernet-based LAN, a single port of a router, or a set of
176 interfaces (physical or logical) of a router. Note that every
177 Observation Point is associated with an Observation Domain
178 (defined below), and that one Observation Point may be a superset
179 of several other Observation Points. For example one Observation
180 Point can be an entire line card. That would be the superset of
181 the individual Observation Points at the line card's interfaces.
183 Observation Domain: An Observation Domain is the largest set of
184 Observation Points for which Flow information can be aggregated by
185 a Metering Process. For example, a router line card may be an
186 Observation Domain if it is composed of several interfaces, each
187 of which is an Observation Point. In the IPFIX Message it
188 generates, the Observation Domain includes its Observation Domain
189 ID, which is unique per Exporting Process. That way, the
190 Collecting Process can identify the specific Observation Domain
191 from the Exporter that sends the IPFIX Messages. Every
192 Observation Point is associated with an Observation Domain. It is
193 RECOMMENDED that Observation Domain IDs are also unique per IPFIX
194 Device.
196 IP Traffic Flow or Flow: There are several definitions of the term
197 'flow' being used by the Internet community. Within the context
198 of IPFIX we use the following definition:
200 A Flow is defined as a set of IP packets passing an Observation
201 Point in the network during a certain time interval. All packets
202 belonging to a particular Flow have a set of common properties.
203 Each property is defined as the result of applying a function to
204 the values of:
206 1. one or more packet header field (e.g. destination IP address),
207 transport header field (e.g. destination port number), or
208 application header field (e.g. RTP header fields [RFC3550])
210 2. one or more characteristics of the packet itself (e.g. number
211 of MPLS labels, etc...)
213 3. one or more of fields derived from packet treatment (e.g. next
214 hop IP address, the output interface, etc...)
216 A packet is defined to belong to a Flow if it completely satisfies
217 all the defined properties of the Flow.
219 This definition covers the range from a Flow containing all
220 packets observed at a network interface to a Flow consisting of
221 just a single packet between two applications. It includes
222 packets selected by a sampling mechanism.
224 Flow Record: A Flow Record contains information about a specific
225 Flow that was observed at an Observation Point. A Flow Record
226 contains measured properties of the Flow (e.g. the total number of
227 bytes for all the Flow's packets) and usually characteristic
228 properties of the Flow (e.g. source IP address).
230 Metering Process: The Metering Process generates Flow Records.
231 Inputs to the process are packet headers and characteristics
232 observed at an Observation Point, and packet treatment at the
233 Observation Point (for example the selected output interface).
235 The Metering Process consists of a set of functions that includes
236 packet header capturing, timestamping, sampling, classifying, and
237 maintaining Flow Records.
239 The maintenance of Flow Records may include creating new records,
240 updating existing ones, computing Flow statistics, deriving
241 further Flow properties, detecting Flow expiration, passing Flow
242 Records to the Exporting Process, and deleting Flow Records.
244 Exporting Process: The Exporting Process sends Flow Records to one
245 or more Collecting Processes. The Flow Records are generated by
246 one or more Metering Processes.
248 Exporter: A device which hosts one or more Exporting Processes is
249 termed an Exporter.
251 IPFIX Device: An IPFIX Device hosts at least one Exporting Process.
252 It may host further Exporting processes and arbitrary numbers of
253 Observation Points and Metering Process.
255 Collecting Process: A Collecting Process receives Flow Records from
256 one or more Exporting Processes. The Collecting Process might
257 process or store received Flow Records, but such actions are out
258 of scope for this document.
260 Template: Template is an ordered sequence of (type, length) pairs,
261 used to completely specify the structure and semantics of a
262 particular set of information that needs to be communicated from
263 an IPFIX Device to a Collector. Each Template is uniquely
264 identifiable by means of a Template ID.
266 Template Record: A Template Record defines the structure and
267 interpretation of fields in a Data Record.
269 Data Record: A Data Record is a record that contains values of the
270 parameters corresponding to a Template Record.
272 Options Template Record: An Options Template Record is a Template
273 Record that defines the structure and interpretation of fields in
274 a Data Record, including defining how to scope the applicability
275 of the Data Record.
277 Set: Set is a generic term for a collection of records that have a
278 similar structure. In an IPFIX Message, one or more Sets follow
279 the Message Header. There are three different types of Sets:
280 Template Set, Options Template Set, and Data Set.
282 Template Set: A Template Set is a collection of one or more
283 Template Records that have been grouped together in an IPFIX
284 Message.
286 Options Template Set: An Options Template Set is a collection of
287 one or more Options Template Records that have been grouped
288 together in an IPFIX Message.
290 Data Set: A Data Set is one or more Data Records, of the same type,
291 that are grouped together in an IPFIX Message. Each Data Record
292 is previously defined by a Template Record or an Options Template
293 Record.
295 Information Element: An Information Element is a protocol and
296 encoding independent description of an attribute which may appear
297 in an IPFIX Record. The IPFIX information model
298 [I-D.ietf-ipfix-info] defines the base set of Information Elements
299 for IPFIX. The type associated with an Information Element
300 indicates constraints on what it may contain and also determines
301 the valid encoding mechanisms for use in IPFIX.
303 Observed Packet Stream: The Observed Packet Stream is the set of
304 all packets observed at the Observation Point.
306 Packet content: The packet content denotes the union of the packet
307 header (which includes link layer, network layer and other
308 encapsulation headers) and the packet payload.
310 Selection Process: A Selection Process takes the Observed Packet
311 Stream as its input and selects a subset of that stream as its
312 output.
314 Selector: A Selector defines the action of a Selection Process on a
315 single packet of its input. If selected, the packet becomes an
316 element of the output Packet Stream.
318 The Selector can make use of the following information in
319 determining whether a packet is selected:
321 1. the Packet Content;
323 2. information derived from the packet's treatment at the
324 Observation Point;
326 3. any selection state that may be maintained by the Selection
327 Process.
329 PSAMP Device: A PSAMP Device is a device hosting at least an
330 Observation Point, a Selection Process and an Exporting Process.
331 Typically, corresponding Observation Point(s), Selection
332 Process(es) and Exporting Process(es) are co-located at this
333 device, for example at a router.
335 Filtering: A filter is a Selector that selects a packet
336 deterministically based on the Packet Content, or its treatment,
337 or functions of these occurring in the Selection State. Examples
338 include field match Filtering, and Hash-based Selection.
340 Transport Session: In SCTP, the transport session is known as the
341 SCTP association, which is uniquely identified by the SCTP
342 endpoints [RFC2960]; in TCP, the transport session is known as the
343 TCP connection, which is uniquely identified by the combination of
344 IP addresses and TCP ports used; In UDP, the transport session is
345 known as the UDP session, which is uniquely identified by the
346 combination of IP addresses and UDP ports used.
348 commonPropertiesID: The commonPropertiesID is an identifier of a
349 set of common properties that is locally unique per Observation
350 Domain and Transport Session. Typically, this Information Element
351 is used to link to information reported in separate Data Records.
352 See the IPFIX information model [I-D.ietf-ipfix-info] for the
353 Information Element definition.
355 Common Properties: Common Properties are a collection of one or
356 more attributes shared by a set of different Flow Records. Each
357 set of Common Properties is uniquely identifiable by means of a
358 commonPropertiesID.
360 Specific Properties: Specific Properties are a collection of one or
361 more attributes reported in a Flow Record that are not included in
362 the Common Properties defined for that Flow Record.
364 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
365 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
366 document are to be interpreted as described in RFC 2119 [RFC2119].
368 2.1. Terminology Summary Table
370 +------------------+---------------------------------------------+
371 | | Contents |
372 | +--------------------+------------------------+
373 | Set | Template | Record |
374 +------------------+--------------------+------------------------+
375 | Data Set | / | Data Record(s) |
376 +------------------+--------------------+------------------------+
377 | Template Set | Template Record(s) | / |
378 +------------------+--------------------+------------------------+
379 | Options Template | Options Template | / |
380 | Set | Record(s) | |
381 +------------------+--------------------+------------------------+
382 Terminology Summary Table
384 A Data Set is composed of Data Record(s). No Template Record is
385 included. A Template Record or an Options Template Record defines
386 the Data Record.
388 A Template Set contains only Template Record(s).
390 An Options Template Set contains only Options Template Record(s).
392 2.2. IPFIX Flows versus PSAMP Packets
394 As described in the PSAMP protocol specification
395 [I-D.ietf-psamp-protocol], the major difference between IPFIX and
396 PSAMP is that the IPFIX protocol exports Flow Records while the PSAMP
397 protocol exports Packet Records. From a pure export point of view,
398 IPFIX will not distinguish a Flow Record composed of several packets
399 aggregated together from a Flow Record composed of a single packet.
400 So the PSAMP export can be seen as special IPFIX Flow Record
401 containing information about a single packet.
403 For this document clarity, the term Flow Record represents a generic
404 term expressing an IPFIX Flow Record or a PSAMP packet record, as
405 foreseen by its definition. However, when appropriate, a clear
406 distinction between Flow Record or packet Record will be made.
408 3. Specifications for bandwidth saving information export
410 Several Flow Records often share a set of Common Properties.
411 Repeating the information about these Common Properties for every
412 Flow Record introduces a huge amount of redundancy. This document
413 proposes a method to reduce this redundancy.
415 The PSAMP specifications are used for the export of per-packet
416 information, exporting the specific observed packet in an IPFIX Flow
417 Record. This can be considered as a special Flow Record case,
418 composed of a single packet. Therefore, the method described in this
419 document is also applicable to per packet data reduction, e.g. for
420 export of One Way Delay (OWD) measurements (see Appendix), trajectory
421 sampling, etc.
423 3.1. Problem Statement and High Level Solution
425 Consider a set of properties "A", e.g. common sourceAddressA and
426 sourcePortA, equivalent for each Flow Records exported. Figure 2
427 shows how this information is repeated with classical IPFIX Flow
428 Records, expressing the waste of bandwidth to export redundant
429 information.
431 +----------------+-------------+---------------------------+
432 | sourceAddressA | sourcePortA | |
433 +----------------+-------------+---------------------------+
434 | sourceAddressA | sourcePortA | |
435 +----------------+-------------+---------------------------+
436 | sourceAddressA | sourcePortA | |
437 +----------------+-------------+---------------------------+
438 | sourceAddressA | sourcePortA | |
439 +----------------+-------------+---------------------------+
440 | ... | ... | ... |
441 +----------------+-------------+---------------------------+
443 Figure 2: Common and Specific Properties exported together
445 Figure 3 shows how this information is exported when applying the
446 specifications of this document. The Common Properties are separated
447 from the Specific Properties for each Flow Record. The Common
448 Properties would be exported only once in a specific Data Record
449 (defined by an Option Template), while each Flow Record contains a
450 pointer to the Common Properties A, along with its Flow specific
451 information. In order to maintain the relationship between these
452 sets of properties, we introduce indices (in this case: index for
453 properties A) for the Common Properties that are unique for all
454 Common Properties entries within an Observation Domain. The purpose
455 of the indices is to serve as a "key" identifying "rows" of the
456 Common Properties table. The rows are then referenced by the
457 Specific Properties by using the appropriate value for the Common
458 Properties identifier.
460 +------------------------+-----------------+-------------+
461 | index for properties A | sourceAddressA | sourcePortA |
462 +------------------------+-----------------+-------------+
463 | ... | ... | ... |
464 +------------------------+-----------------+-------------+
466 +------------------------+---------------------------+
467 | index for properties A | |
468 +------------------------+---------------------------+
469 | index for properties A | |
470 +------------------------+---------------------------+
471 | index for properties A | |
472 +------------------------+---------------------------+
473 | index for properties A | |
474 +------------------------+---------------------------+
475 Figure 3: Common and Specific Properties exported separately
477 This unique export of the Common Properties results in a decrease of
478 the bandwidth requirements for the path between the Exporter and the
479 Collector.
481 3.2. Data Reduction technique
483 The IPFIX protocol [I-D.ietf-ipfix-protocol] is Template based.
484 Templates define how data should be exported, describing data fields
485 together with their type and meaning. IPFIX specifies two types of
486 Templates: the Template Record and the Options Template Record. The
487 difference between the two is that the Options Template Record
488 includes the notion of scope, defining how to scope the applicability
489 of the Data Record. The scope, which is only available in the
490 Options Template Record, gives the context of the reported
491 Information Elements in the Data Records. The Template Records and
492 Options Template Records are necessary to decode the Data Records.
493 Indeed, by only looking at the Data Records themselves, this is
494 impossible to distinguish a Data Record defined by Template Record
495 from a Data Record defined by an Option Template Record. To export
496 information more efficiently, this specification proposes to group
497 Flow Records by their common properties. We define Common Properties
498 as a collection of attributes shared by a set of different Flow
499 Records.
501 An implementation using the proposed specification MUST follow the
502 IPFIX transport protocol specifications defined in the IPFIX protocol
503 [I-D.ietf-ipfix-protocol].
505 As explained in Figure 4, the information is split into two parts,
506 using two different Data Records. Common Properties MUST be exported
507 via Data Records defined by an Option Template Record. Like Template
508 Records, they MUST be sent only once per SCTP association or TCP
509 connection, and MUST be sent reliably via SCTP if SCTP is the
510 transport protocol. These properties represent values common to
511 several Flow Records (e.g. IP source and destination address). The
512 Common Properties Data Records MUST be sent prior to the
513 corresponding Specific Properties Data Records. The Data Records
514 reporting Specific Properties MUST be associated with the Data
515 Records reporting the Common Properties using a unique identifier for
516 the Common Properties, the commonPropertiesID Information Element
517 [I-D.ietf-ipfix-info]. The commonPropertiesID MUST be exported as
518 the scope in the Options Template Record, and also exported in the
519 associated Template Record.
521 +---------------------------+ +---------------------+
522 | Common Properties | | Specific Properties | Template
523 | Option Template Record | | Template Record | Definition
524 | | | |
525 | scope: commonPropertiesID | | commonPropertiesID |
526 | Common Properties | | Specific Properties |
527 +------------+--------------+ +---------+-----------+
528 .............|...............................|.......................
529 | |
530 +------------v-------------+ +----------v----------+
531 | Common Properties | | Specific Properties |+ Exported
532 | Data Record |------> Data Records || Data
533 +--------------------------+ +---------------------+| Records
534 +---------------------+
536 Figure 4: Template Record and Data Record dependencies
538 From the IPFIX protocol, there are no differences between the per
539 Flow or per packet data reduction, except maybe the terminology where
540 the Specific Properties could be called packet Specific Properties in
541 the previous figure.
543 4. Transport Protocol Choice
545 This document follows the IPFIX transport protocol specifications
546 defined in the IPFIX protocol [I-D.ietf-ipfix-protocol]. However,
547 depending on the transport protocol choice, this document imposes
548 some more constraints. If PR-SCTP is selected as the IPFIX protocol,
549 the SCTP sub-section specifications MUST be respected. If UDP is
550 selected as the IPFIX protocol, the UDP sub-section specifications
551 MUST be respected. If TCP is selected as the IPFIX protocol, the TCP
552 sub-section specifications MUST be respected.
554 4.1. PR-SCTP
556 The active Common Properties MUST be sent after the SCTP association
557 establishment before the corresponding Specific Properties Data
558 Records. In case of SCTP association re-establishment, all active
559 Common Properties MUST be re-sent before the corresponding Specific
560 Properties Data Records.
562 The Common Properties Data Records MUST be sent reliably.
564 4.2. UDP
566 Common Properties Data Records MUST be re-sent at regular intervals,
567 whose frequency MUST be configurable. The default value for the
568 frequency of Common Properties transmission (refresh timeout) is 10
569 minutes.
571 The Exporting Process SHOULD transmit the Common Properties
572 definition in advance of any Data Record that use these Common
573 Properties, to help ensure that the Collector has the Common
574 Properties definition before receiving the first associated Data
575 Record.
577 If a commonPropertiesID is not used anymore the Exporting Process
578 stops re-sending the related Common Properties Data Record. The old
579 commonPropertiesID MUST NOT be used until its lifetime (see
580 Section 6.1) has expired.
582 4.3. TCP
584 Common Properties MUST be sent after the TCP connection establishment
585 before the corresponding Specific Properties Data Records. In case
586 of TCP connection re-establishment, all active Common Properties MUST
587 be re-sent before the corresponding Specific Properties Data Records.
589 5. commonPropertiesID Management
591 The commonPropertiesId is an identifier of a set of common properties
592 that is locally unique per Observation Domain and Transport Session.
593 The Exporting Process MUST manage the commonPropertiesIDs allocations
594 for its Observation Domains and Transport Session. Different
595 Observation Domains from the same Exporter MAY use the same
596 commonPropertiesID value to refer to different sets of Common
597 Properties.
599 The commonPropertiesID values MAY be assigned sequentially, but it is
600 NOT REQUIRED. Particular commonPropertiesID ranges or values MAY
601 have explicit meanings for the IPFIX Device. For example,
602 commonPropertiesID values may be assigned based on the result of a
603 hash function, etc...
605 Using a 64 bit commonPropertiesID Information Element allows the
606 export of 2**64 active sets of Common Properties, per Observation
607 Domain and per Transport Session.
609 commonPropertiesIDs that are not used anymore SHOULD be withdrawn.
610 The Common Properties ID withdrawal message is a Data Record defined
611 by an Option Template consisting of only one scope field namely the
612 commonPropertiesID (with a type of 137 [I-D.ietf-ipfix-info]) and no
613 non-scope fields.
615 0 1 2 3
616 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
617 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
618 | Set ID = 3 | Length = 14 octets |
619 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
620 | Template ID N | Field Count = 1 |
621 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
622 | Scope Field count = 1 |0| commonPropertiesID = 137 |
623 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
624 | Scope 1 Field Length = 8 |
625 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
627 commonPropertiesID Withdrawal Message
629 If UDP is selected as the transport protocol, the commonPropertiesID
630 Template Withdraw Messages MUST not be used, as this method is
631 inefficient due to the unreliable nature of UDP.
633 6. The Collecting Process Side
635 This section describes the Collecting Process when using SCTP and PR-
636 SCTP as the transport protocol. Any necessary changes to the
637 Collecting Process specifically related to TCP or UDP transport
638 protocols are specified in the subsections.
640 The Collecting Process MUST store the commonPropertiesId information
641 for the duration of the association so that it can interpret the
642 corresponding Data Records that are received in subsequent Data Sets.
643 The Collecting Process can either store the Data Records as they
644 arrive, without reconstructing the initial Flow Record, or
645 reconstruct the initial Flow Record. In the former case, there might
646 be less storage capacity required at the Collector side. In the
647 latter case, the collector job is more complex and time-consuming due
648 to the higher resource demand for record processing in real time.
650 If the Collecting Process has received the Specific Properties Data
651 Record before the associated Common Properties Data Record, the
652 Collecting Process SHOULD store the Specific Properties Data Record
653 and await the retransmission or out-of-order arrival of the Common
654 Properties Data Record.
656 Common Properties IDs are unique per SCTP association and per
657 Observation Domain. If the Collecting Process receives a Common
658 Properties ID which has already been received but which has not
659 previously been withdrawn (i.e. a commonPropertiesID from the same
660 Exporter Observation Domain received on the SCTP association), then
661 the Collecting Process MUST shutdown the association.
663 When an SCTP association is closed, the Collecting Process MUST
664 discard all Common Properties IDs received over that association and
665 stop decoding IPFIX Messages that use those Common Properties IDs.
667 If a Collecting Process receives a Common Properties Withdrawal
668 message, the Collecting Process MUST delete the corresponding Common
669 Properties associated with the specific SCTP association and specific
670 Observation Domain, and stop interpreting Data Records referring to
671 those Common Properties. The receipt of Data Records referring to
672 Common Properties that have been withdrawn MUST be ignored and SHOULD
673 be logged by the Collecting Process.
675 If the Collecting Process receives a Common Properties Withdrawal
676 message for a Common Properties that it has not received before on
677 this SCTP assocation, it MUST reset the SCTP association, discard the
678 IPFIX Message, and SHOULD log the error as it does for malformed
679 IPFIX Messages.
681 6.1. UDP
683 The Collecting Process MUST associate a lifetime with each Common
684 Property received via UDP. Common Properties not refreshed by the
685 Exporting Process within the lifetime are expired at the Collecting
686 Process.
688 If the Common Properties are not refreshed before that lifetime has
689 expired, the Collecting Process MUST discard the corresponding
690 definition of the commonPropertiesID and any current and future
691 associated Data Records. In which case, an alarm MUST be logged.
693 The Collecting Process MUST NOT decode any further Data Records which
694 are associated with the expired Common Properties. If a Common
695 Property is refreshed with a definition that differs from the
696 previous definition, the Collecting Process SHOULD log a warning and
697 replace the previous received Common Property with the new one. The
698 Common Property lifetime at the Collecting Process MUST be at least 3
699 times higher than the refresh timeout of the Template used to export
700 the Common Property definition, configured on the Exporting Process.
702 The Collecting Process SHOULD accept Data Records without the
703 associated Common Properties required to decode the Data Record. If
704 the Common Properties have not been received at the time Data Records
705 are received, the Collecting Process SHOULD store the Data Records
706 for a short period of time and decode them after the Common
707 Properties definitions are received. The short period of time MUST
708 be lower than the lifetime of definitions associated with identifiers
709 considered unique within the UDP session.
711 6.2. TCP
713 When the TCP connection is reset, either gracefully or abnormally,
714 the Collecting Processes MUST delete all commonPropertiesID values
715 and associated Common Properties data corresponding to that
716 connection.
718 If a Collection Process receives a commonPropertiesID Withdraw
719 message, the Collection Process MUST expire the related Common
720 Properties data.
722 7. Advanced Techniques
724 7.1. Multiple Data Reduction
726 A Flow Record can refer to one or more Common Properties sets; the
727 use of multiple Common Properties can lead to more efficient exports.
728 When sets of Common Properties are identified in the data, it may be
729 found that there is more than one set of non-overlapping properties.
731 Note that in the case of multiple Common Properties in one Data
732 Record, the different sets of Common Properties MUST be disjoint
733 (i.e. MUST NOT have Information Elements in common), to avoid
734 potential collisions.
736 Consider a set of properties "A", e.g. common sourceAddressA and
737 sourcePortA, and another set of properties "B", e.g.
738 destinationAddressB and destinationPortB. Figure 6 shows how this
739 information is repeated with classical IPFIX export in several Flow
740 Records.
742 +--------+--------+---------+---------+---------------------+
743 |srcAddrA|srcPortA|destAddrB|destPortB| |
744 +--------+--------+---------+---------+---------------------+
745 |srcAddrA|srcPortA|destAddrC|destPortC| |
746 +--------+--------+---------+---------+---------------------+
747 |srcAddrD|srcPortD|destAddrB|destPortB| |
748 +--------+--------+---------+---------+---------------------+
749 |srcAddrD|srcPortD|destAddrC|destPortC| |
750 +--------+--------+---------+---------+---------------------+
751 | ... | ... | ... | ... | ... |
752 +--------+--------+---------+---------+---------------------+
754 Figure 6: Common and Specific Properties exported together
756 Besides that other sets of Properties might be repeated as well (e.g.
757 properties C and D in the figure above).
759 We can separate the Common Properties into the properties A composed
760 of sourceAddressA and sourcePortA, properties D composed of
761 sourceAddressD and sourcePortD, and into the properties B composed of
762 destinationAddressB and destinationPortB and properties C composed of
763 destinationAddressC and destinationPortC,. These four records can be
764 expanded to four combinations of Data Records to reduce redundancy
765 without the need to define four complete sets of Common Properties
766 (see the figure below). The more Common Properties sets are defined,
767 the more combinations are available.
769 +-------------------+-----------------+-------------+
770 | index for prop. A | sourceAddressA | sourcePortA |
771 +-------------------+-----------------+-------------+
772 | index for prop. D | sourceAddressD | sourcePortD |
773 +-------------------+-----------------+-------------+
775 +-------------------+---------------------+------------------+
776 | index for prop. B | destinationAddressB | destinationPortB |
777 +-------------------+---------------------+------------------+
778 | index for prop. C | destinationAddressC | destinationPortC |
779 +-------------------+---------------------+------------------+
781 +-----------------+-----------------+-----------------------+
782 |index for prop. A|index for prop. B| |
783 +-----------------+-----------------+-----------------------+
784 |index for prop. A|index for prop. C| |
785 +-----------------+-----------------+-----------------------+
786 |index for prop. D|index for prop. B| |
787 +-----------------+-----------------+-----------------------+
788 |index for prop. D|index for prop. C| |
789 +-----------------+-----------------+-----------------------+
791 Multiple Common (above) and Specific Properties (below) exported
792 separately
794 The advantage of the multiple Common Properties is that the objective
795 of reducing the bandwidth is met while the number of indexes is kept
796 to a minimum. Defining an extra index for all records would not
797 bring to save bandwidth in the case of Figure 6 and is generally a
798 less efficient solution.
800 If a set of Flow Records share multiple sets of Common Properties,
801 multiple commonPropertiesID instances MAY be used to increase export
802 efficiency even further, as displayed in Figure 8.
804 +--------------------------- + +---------------------+
805 | Common Properties | | Specific Properties | Template
806 | Option Template Record | | Template Record | Definition
807 | | | |
808 | Scope: commonPropertiesID1 | | commonPropertiesID1 |
809 | Scope: commonPropertiesID2 | | commonPropertiesID2 |
810 | Common Properties | | Specific Properties |
811 +------------+---------------+ +--------+------------+
812 .............|...............................|.......................
813 | |
814 +------------v-------------+ +----------v----------+
815 | Common Properties | | Specific Properties |+ Exported
816 | Data Record |------> Data Records || Data
817 +------------------------- + +---------------------+| Records
818 +---------------------+
820 Figure 8: Multiple Data Reduction
822 7.2. Cascading Common Properties
824 An Exporting Process MUST NOT export any set of Common Properties
825 which contains, either directly or via other cascaded Common
826 Properties, references to itself in its own definition (i.e., a
827 circular definition). When the Collecting Process receives Common
828 Properties that reference other Common Properties, it MUST resolve
829 the references to Common Properties. If the Common Properties aren't
830 available at the time Data Records are received, the Collecting
831 Process SHOULD store the Data Records for a short period of time and
832 decode them after the Common Properties are received.
834 If the Collecting Process could not decode a cascading Common
835 Properties definition because the referenced Common Properties are
836 not available before the short period of time, then the Collecting
837 Process SHOULD log the error.
839 If the Collecting Process could not decode a cascading Common
840 Properties definition because it detects a circular definition, then
841 the Collecting Process SHOULD log the error.
843 Information Element ordering MUST be preserved when creating and
844 expanding Common Properties.
846 8. Export and Evaluation Considerations
848 The objective of the method specified in this document is the
849 reduction in the amount of measurement data that has to be
850 transferred from the Exporter to the Collector. Note that the
851 efficiency of this method may vary, as discussed in this section. In
852 addition there might be less storage capacity required at the
853 Collector side if the Collector decides to store the Data Records as
854 they arrive, without reconstructing the initial Flow Record.
856 On the other hand, this method requires additional resources on both
857 the Exporter and the Collector. The Exporter has to manage Common
858 Properties information and to assign commonPropertiesId values. The
859 Collector has to process records described by two templates instead
860 of just one. Additional effort is also required when post processing
861 the measurement data, in order to correlate Flow Records with Common
862 Properties information.
864 8.1. Transport Protocol Choice
866 The proposed method is most effective using a reliable transport
867 protocol for the transfer of the Common Properties. Therefore the
868 use of PR-SCTP with the reliable mode or TCP is recommended.
869 However, if the path from the Exporting Process to the Collecting
870 Process is not fully reliable, the SCTP or TCP retransmission might
871 reduce the benefits of this specification. If the path from the
872 Exporting Process to the Collecting Process is full reliable, the use
873 of UDP is less effective because the Common Properties have to be re-
874 sent regularly.
876 8.2. Reduced Size Encoding
878 The transfer of the commonPropertiesIDs originates some overhead and
879 might even increase the amount of exported data if the length of the
880 commonPropertiesID field is not shorter than the length of the
881 replaced fields.
883 In cases where the range of the commonPropertiesID can be restricted,
884 it is RECOMMENDED to apply reduced-size encoding to the
885 commonPropertiesID, to achieve a further bandwidth efficiency gain.
887 8.3. Efficiency Gain
889 While the goal of this specification is to reduce the bandwidth, the
890 efficiency might be limited. Indeed, the efficiency gain is based on
891 the numerous redundant information in Flows and would be directly
892 proportional to the re-use of the defined commonPropertiesID values
893 (In other words, the more we re-use a commonPropertiesID value, the
894 better the efficiency gain), with a theoretical limit where all the
895 Data Records would use a single commonPropertiesID. While the
896 Exporting Process can evaluate the direct gain for the Flow Records
897 to be exported, it cannot predict whether future Flow Records would
898 contain the information specified by active commonPropertiesID
899 values. This implies that the efficiency factor of this
900 specification is higher for specific applications where filtering is
901 involved, such as one-way delay or trajectory sampling.
903 Note that this technique might even lead to an increase in bandwidth
904 usage under certain conditions. Taking into account the overhead of
905 exporting the commonPropertiesID values, if the commonPropertiesID
906 values are not used in future Data Records, this technique would
907 actually increase the export bandwidth. A typical case would be the
908 assignments of Common Properties based on past observed traffic,
909 hoping that future Flows would contain the same characteristics.
911 The efficiency gain depends also on the difference between the length
912 of the replaced fields and the length of the commonPropertiesID. The
913 shorter is the length of commonPropertiesID with respect to the total
914 length of the Common Properties fields, the bigger is the gain.
916 The example in section Appendix A.2 below uses IPFIX to export
917 measurement data for each received packet. In that case, for a Flow
918 of 1000 packets the amount of data can be decreased more than 26
919 percent.
921 9. IANA Considerations
923 This document has no actions for IANA.
925 10. Security Considerations
927 The same security considerations as for the IPFIX Protocol
928 [I-D.ietf-ipfix-protocol] apply.
930 11. Acknowledgments
932 The authors would like to thank Guido Pohl for initiating this work
933 and for his contribution to early versions of this document. Thanks
934 also to Andrew Johnson, Gehrard Muenz, Brian Trammell and Paul Aitken
935 for their comments and feedback.
937 12. References
938 12.1. Normative References
940 [I-D.ietf-ipfix-protocol]
941 Claise, B., "Specification of the IPFIX Protocol for the
942 Exchange", draft-ietf-ipfix-protocol-24 (work in
943 progress), November 2006.
945 [I-D.ietf-ipfix-info]
946 Quittek, J., "Information Model for IP Flow Information
947 Export", draft-ietf-ipfix-info-15 (work in progress),
948 February 2007.
950 [I-D.ietf-psamp-protocol]
951 Claise, B., "Packet Sampling (PSAMP) Protocol
952 Specifications", draft-ietf-psamp-protocol-07 (work in
953 progress), October 2006.
955 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
956 Requirement Levels", BCP 14, RFC 2119, March 1997.
958 12.2. Informative References
960 [I-D.ietf-ipfix-as]
961 Zseby, T., "IPFIX Applicability", draft-ietf-ipfix-as-11
962 (work in progress), February 2007.
964 [I-D.ietf-ipfix-architecture]
965 Sadasivan, G., "Architecture for IP Flow Information
966 Export", draft-ietf-ipfix-architecture-12 (work in
967 progress), September 2006.
969 [I-D.ietf-psamp-info]
970 Dietz, T., "Information Model for Packet Sampling
971 Exports", draft-ietf-psamp-info-05 (work in progress),
972 October 2006.
974 [I-D.ietf-psamp-sample-tech]
975 Zseby, T., "Sampling and Filtering Techniques for IP
976 Packet Selection", draft-ietf-psamp-sample-tech-07 (work
977 in progress), July 2005.
979 [I-D.ietf-psamp-mib]
980 Dietz, T. and B. Claise, "Definitions of Managed Objects
981 for Packet Sampling", draft-ietf-psamp-mib-06 (work in
982 progress), June 2006.
984 [I-D.ietf-psamp-framework]
985 Duffield, N., "A Framework for Packet Selection and
986 Reporting", draft-ietf-psamp-framework-11 (work in
987 progress), May 2007.
989 [RFC2960] Stewart, R., Xie, Q., Morneault, K., Sharp, C.,
990 Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M.,
991 Zhang, L., and V. Paxson, "Stream Control Transmission
992 Protocol", RFC 2960, October 2000.
994 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
995 Jacobson, "RTP: A Transport Protocol for Real-Time
996 Applications", STD 64, RFC 3550, July 2003.
998 [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander,
999 "Requirements for IP Flow Information Export (IPFIX)",
1000 RFC 3917, October 2004.
1002 Appendix A. Examples
1004 A.1. Per Flow Data Reduction
1006 In this section we show how Flow information can be exported
1007 efficiently using the method described in this draft. Let's suppose
1008 we have to periodically export data about two IPv6 Flows.
1010 In this example we report the following information:
1012 Flow| dstIPv6Address | dst- |nPkts|nBytes
1013 | | Port | |
1014 ----------------------------------------------------------------
1015 A |2001:DB8:80AD:5800:0058:0800:2023:1D71 | 80 | 30 | 6000
1016 | | | |
1017 A |2001:DB8:80AD:5800:0058:0800:2023:1D71 | 80 | 50 | 9500
1018 | | | |
1019 B |2001:DB8:80AD:5800:0058:00AA:00B7:AF2B | 1932 | 60 | 8000
1020 | | | |
1021 A |2001:DB8:80AD:5800:0058:0800:2023:1D71 | 80 | 40 | 6500
1022 | | | |
1023 A |2001:DB8:80AD:5800:0058:0800:2023:1D71 | 80 | 60 | 9500
1024 | | | |
1025 B |2001:DB8:80AD:5800:0058:00AA:00B7:AF2B | 1932 | 54 | 7600
1027 The Common Properties in this case are the destination IPv6 address
1028 and the destination port. We first define an Option Template that
1029 contains the following Information Elements:
1031 o Scope: commonPropertiesID in [I-D.ietf-ipfix-info], with a type of
1032 137 and a length of 8 octets.
1034 o The destination IPv6 address: destinationIPv6Address in
1035 [I-D.ietf-ipfix-info], with a type of 28 and a length of 16
1036 octets.
1038 o The destination port: destinationTransportPort in
1039 [I-D.ietf-ipfix-info], with a type of 11, and a length of 2 octets
1041 Figure 10 shows the Option template defining the Common Properties
1042 with commonPropertiesID as scope:
1044 0 1 2 3
1045 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1046 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1047 | Set ID = 3 | Length = 24 octets |
1048 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1049 | Template ID = 257 | Field Count = 3 |
1050 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1051 | Scope Field count = 1 |0| commonPropertiesID = 137 |
1052 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1053 | Scope 1 Field Length = 8 |0| destinationIPv6Address = 28|
1054 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1055 | Field Length = 16 |0|destinationTransportPort = 11|
1056 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1057 | Field Length = 2 | (Padding) |
1058 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1060 Figure 10: Common Properties Option Template
1062 The Specific Properties Template consists of the information not
1063 contained in the Option Templates, i.e. Flow specific information, n
1064 this case the number of packets and the number of bytes to be
1065 reported. Additionally, this Template contains the
1066 commonPropertiesID. In Data Records, the value of this field will
1067 contain one of the unique indices of the Option Records exported
1068 before. It contains the following Information Elements (see also
1069 Figure 11):
1071 o commonPropertiesID with a length of 8 octets
1073 o The number of packets of the Flow: inPacketDeltaCount in
1074 [I-D.ietf-ipfix-info], with a length of 4 octets.
1076 o The number of octets of the Flow: inOctetDeltaCount in
1077 [I-D.ietf-ipfix-info], with a length of 4 octets
1079 0 1 2 3
1080 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1081 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1082 | Set ID = 2 | Length = 20 octets |
1083 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1084 | Template ID = 258 | Field Count = 3 |
1085 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1086 |0| commonPropertiesID = 137 | Field Length = 8 |
1087 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1088 |0| inPacketDeltaCount = 2 | Field Length = 4 |
1089 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1090 |0| inOctetDeltaCount = 1 | Field Length = 4 |
1091 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1093 Figure 11: Specific Properties Template
1095 Considering the data shown at the beginning of this example, the
1096 following two Data Records will be exported:
1098 Common- | dstAddress | dst-
1099 PropertiesID | | Port
1100 -------------+-----------------------------------------+-------
1101 101 | 2001:DB8:80AD:5800:0058:0800:2023:1D71 | 80
1102 | |
1103 102 | 2001:DB8:80AD:5800:0058:00AA:00B7:AF2B | 1932
1105 The Data Records reporting the Common Properties will look like:
1107 0 1 2 3
1108 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1109 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1110 | Set ID = 257 | Length = 60 octets |
1111 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1112 | |
1113 +- 101 -+
1114 | |
1115 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1116 | |
1117 +- -+
1118 | |
1119 +- 2001:DB8:80AD:5800:0058:0800:2023:1D71 -+
1120 | |
1121 +- -+
1122 | |
1123 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1124 | 80 | |
1125 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- -+
1126 | 102 |
1127 +- -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1128 | | |
1129 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -+
1130 | |
1131 +- -+
1132 | 2001:DB8:80AD:5800:0058:00AA:00B7:AF2B |
1133 +- -+
1134 | |
1135 +- -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1136 | | 1932 |
1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1139 Figure 13: Data Records reporting Common Properties
1141 The Data Records will in turn be:
1143 commonPropertiesID | inPacketDeltaCount | inOctetDeltaCount
1144 ---------------------------------------------------------------
1145 101 | 30 | 6000
1146 101 | 50 | 9500
1147 102 | 60 | 8000
1148 101 | 40 | 6500
1149 101 | 60 | 9500
1150 102 | 54 | 7600
1152 Figure 15 shows the first Data Record listed in the table:
1154 0 1 2 3
1155 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1156 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1157 | Set ID = 258 | Length = 16 |
1158 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1159 | |
1160 +- 101 -+
1161 | |
1162 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1163 | 30 | 6000 |
1164 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1166 Figure 15: Data Records reporting Common Properties
1168 A.2. Per Packet Data Reduction
1170 An example of the per packet data reduction is the measurement of
1171 One-Way Delay (OWD), where the exact same specific packet must be
1172 observed at the source and destination of the path to be measured.
1173 By subtracting the time of observation of the same packet at the two
1174 end points with synchronized clocks, the OWD is computed. As the OWD
1175 is measured for a specific application on which a Service Level
1176 Agreement (SLA) is bound, this translates into the observation of
1177 multiple packets with Specific Properties, results of filtering. In
1178 order to match the identical packet at both Observation Points, a
1179 series of packets with a set of properties (For example, all the
1180 packets of a specific source and destination IP addresses, of a
1181 specific DSCP value, and of a specific destination transport port)
1182 must be observed at both ends of the measurements. This implies that
1183 the source and destination must export of a series of Flow Records
1184 composed of two types of information: some common information for all
1185 packets, and some unique information about packets in order to
1186 generate a unique identifier for each packet passing this Observation
1187 Point (for example, a hash value on the invariant fields of the
1188 packet). So, the source and destination composing the measurements
1189 end points can individually and independently apply the redundancy
1190 technique described in this draft in order to save some bandwidth for
1191 their respective Flow Records export.
1193 The Templates required for exporting measurement data of this kind
1194 are illustrated in the figures below. Figure 16 shows the Option
1195 Template containing the information concerning Flows using the
1196 commonPropertiesID as scope. In the Common Properties Template we
1197 export the following Information Elements:
1199 o The source IPv4 Address: sourceIPv4Address in
1200 [I-D.ietf-ipfix-info], with a type of 8 and a length of 4 octets.
1202 o The destination IPv4 Address: destinationIPv4Address in
1203 [I-D.ietf-ipfix-info], with a type of 12 and a length of 4 octets.
1205 o The Class of Service field: ClassOfServiceIPv4 in
1206 [I-D.ietf-ipfix-info], with a type of 5 and a length of 1 octet
1208 o The Protocol Identifier: protocolIdentifier in
1209 [I-D.ietf-ipfix-info], with a type of 4 and a length of 1 octet
1211 o The source port: sourceTransportPort in [I-D.ietf-ipfix-info],
1212 with a type of 7 and a length of 2 octets.
1214 o The destination port: destinationTransportPort in
1215 [I-D.ietf-ipfix-info], with a type of 11 and a length of 2 octets.
1217 The commonPropertiesID Information Element is used as the Scope
1218 Field.
1220 0 1 2 3
1221 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1222 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1223 | Set ID = 3 | Length = 40 octets |
1224 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1225 | Template ID = 256 | Field Count = 7 |
1226 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1227 | Scope Field count = 1 |0| commonPropertiesID = 137 |
1228 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1229 | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 |
1230 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1231 | Field Length = 4 |0| destinationIPv4Address = 12 |
1232 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1233 | Field Length = 4 |0| classOfServiceIPv4 = 5 |
1234 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1235 | Field Length = 1 |0| protocolIdentifier = 4 |
1236 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1237 | Field Length = 1 |0| transportSourcePort = 7 |
1238 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1239 | Field Length = 2 |0|transportDestinationPort = 11|
1240 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1241 | Field Length = 2 |
1242 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1244 Figure 16: Example Flow Properties Template
1246 For passive One Way Delay measurement, the Packet Properties
1247 Template, or Specific Properties Template, consists of at least
1248 Timestamp and Packet ID. Additionally, this template contains a
1249 commonPropertiesId field to associate the packet with a Flow.
1251 Figure 17 displays the template with the packet properties. In this
1252 example we export the following Information Elements:
1254 o commonPropertiesID. In this case reduced size encoding is used,
1255 and the Information Element is declared with a length of 4 octets
1256 instead of 8.
1258 o The packet timestamp: observationTimeMilliseconds in the PSAMP
1259 Information Model [I-D.ietf-psamp-info], with a type of 323 and a
1260 length of 8 octets.
1262 o digestHashValue in the PSAMP Information Model
1263 [I-D.ietf-psamp-info], with a type of 326 and a length of 8 octets
1265 o The packet length: ipTotalLength in the IPFIX Information Model
1266 [I-D.ietf-ipfix-info], with a type of 224 and a length of 8 octets
1268 0 1 2 3
1269 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1270 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1271 | Set ID = 2 | Length = 36 octets |
1272 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1273 | Template ID = 257 | Field Count = 4 |
1274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1275 |0| commonPropertiesID = 137 | Field Length = 4 |
1276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1277 |0| observationTimeMillis.= 323 | Field Length = 8 |
1278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1279 |0| digestHashValue = 326 | Field Length = 8 |
1280 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1281 |0| ipTotalLength = 224 | Field Length = 8 |
1282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1284 Figure 17: Example Packet Properties Template
1286 At the collection point, packet records from the two measurement
1287 points are gathered and correlated by means of the packet ID. The
1288 resulting delay data records are exported in a similar manner as the
1289 packet data. One-way delay data is associated with Flow information
1290 by the commonPropertiesId field. The OWD properties contain the
1291 Packet Pair ID (which is the packet ID of the two contributing packet
1292 records), the timestamp of the packet passing the reference monitor
1293 point in order to reconstruct a time series, the calculated delay
1294 value, and the commonPropertiesID.
1296 In this example using IPFIX to export the measurement data for each
1297 received packet 38 bytes have to be transferred (sourceAddressV4=4,
1298 destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1,
1299 sourceTransportPort=2, destionationTransportPort=2,
1300 observationTimeMilliseconds=8, digestHashValue=8, ipTotalLength=8).
1301 Without considering the IPFIX protocol overhead a Flow of 1000
1302 packets produces 38000 bytes of measurement data. Using the proposed
1303 optimization each packet produces an export of only 28 bytes
1304 (observationTimeMilliseconds=8, digestHashValue=8, ipTotalLength=8,
1305 commonPropertiesID=4). The export of the Flow information produces
1306 18 bytes (sourceAddressV4=4, destinationAddressV4=4,
1307 classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2,
1308 destionationTransportPort=2, commonPropertiesID=4). For a Flow of
1309 1000 packets this sums up to 28018 bytes. This is a decrease of more
1310 than 26 percent.
1312 A.3. commonPropertiesID Template Withdrawal Message
1314 This section shows an example commonPropertiesID Withdrawal message.
1315 Figure 18 depicts the Option Template Record with the
1316 commonPropertiesID as unique scope field, and no non-scope fields.
1318 0 1 2 3
1319 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1320 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1321 | Set ID = 3 | Length = 14 octets |
1322 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1323 | Template ID 259 | Field Count = 1 |
1324 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1325 | Scope Field count = 1 |0| commonPropertiesID 137 |
1326 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1327 | Scope 1 Field Length = 8 |
1328 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1330 Figure 18: example commonPropertiesID withdrawal template
1332 Figure 19 shows the Option Data Record withdrawing commonPropertiesID
1333 N:
1335 0 1 2 3
1336 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
1337 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1338 | Set ID = 259 | Length = 12 octets |
1339 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1340 | |
1341 +- N -+
1342 | |
1343 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1345 Figure 19: commonPropertiesID withdrawal record, withdrawing
1346 commonPropertiesID N
1348 Authors' Addresses
1350 Elisa Boschi
1351 Hitachi Europe SAS
1352 Immeuble Le Theleme
1353 1503 Route les Dolines
1354 06560 Valbonne
1355 France
1357 Phone: +33 4 89874100
1358 Email: elisa.boschi@hitachi-eu.com
1360 Lutz Mark
1361 Fraunhofer FOKUS
1362 Kaiserin Augusta Allee 31
1363 10589 Berlin
1364 Germany
1366 Phone: +49 30 34637306
1367 Email: mark@fokus.fraunhofer.de
1369 Benoit Claise
1370 Cisco Systems
1371 De Kleetlaan 6a b1
1372 Diegem 1813
1373 Belgium
1375 Phone: +32 2 704 5622
1376 Email: bclaise@cisco.com
1378 Full Copyright Statement
1380 Copyright (C) The IETF Trust (2007).
1382 This document is subject to the rights, licenses and restrictions
1383 contained in BCP 78, and except as set forth therein, the authors
1384 retain all their rights.
1386 This document and the information contained herein are provided on an
1387 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
1388 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
1389 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
1390 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
1391 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
1392 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
1394 Intellectual Property
1396 The IETF takes no position regarding the validity or scope of any
1397 Intellectual Property Rights or other rights that might be claimed to
1398 pertain to the implementation or use of the technology described in
1399 this document or the extent to which any license under such rights
1400 might or might not be available; nor does it represent that it has
1401 made any independent effort to identify any such rights. Information
1402 on the procedures with respect to rights in RFC documents can be
1403 found in BCP 78 and BCP 79.
1405 Copies of IPR disclosures made to the IETF Secretariat and any
1406 assurances of licenses to be made available, or the result of an
1407 attempt made to obtain a general license or permission for the use of
1408 such proprietary rights by implementers or users of this
1409 specification can be obtained from the IETF on-line IPR repository at
1410 http://www.ietf.org/ipr.
1412 The IETF invites any interested party to bring to its attention any
1413 copyrights, patents or patent applications, or other proprietary
1414 rights that may cover technology that may be required to implement
1415 this standard. Please address the information to the IETF at
1416 ietf-ipr@ietf.org.
1418 Acknowledgment
1420 Funding for the RFC Editor function is provided by the IETF
1421 Administrative Support Activity (IASA).