10 Abstract Current uses of the Internet are mostly built on request-response exchanges between pairs of computers. This model has been very successful at providing shared remote access to resources and point-to-point communications for applications including , remote file and database access, web access, etc. The advent of ubiquitous Internet connectivity and cheap network-connected devices has greatly expanded the communication needs of applications using the Internet. For these new kinds of applications the request-response communications model, which is highly coupled to the location of the information, is cumbersome and limited whereas other communication models, such as publish-subscribe, provide a more natural fit. In contrast to the request-response interaction model, in which a client sends a request and a server returns a response, in the publish-subscribe model information producers (publishers) are decoupled from information consumers (subscribers) via the publish-subscribe service. One of these services is Data Distribution Service (DDS), an standardized solution for data-centric publish-subscribe data distribution. This Thesis describes the design and evaluation of a set of discovery and signaling enhancements for Data-Centric Publish-Subscribe (DCPS) systems. In particular, this Thesis focus on solving the following open issues of DDS: data-spaces interconnection, data transformation, Quality of Service (QoS) adaptation, universal DCPS entity and data-content naming, end-to-end session establishment, and discovery scalability in large deployments. We have divided our solution in three parts. First, we propose the DDS data-space Interconnection Service (DDS-IS), a fully DDS-compliant service for DDS data-spaces interconnection which also provides transparent data transformation and QoS adaptation. We demonstrate that the DDS-IS can improve the scalability of DDS systems by reducing the network traffic load between the interconnected data-spaces. Secondly, we define a mechanism for universal identification of DCPS entities and data-content, and we propose the rationale and design of a protocol for session signaling in DCPS environments. This protocol allows applications to perform DCPS entity and data-content discovery in medium-scale environments. It also allows the creation of sessions between nodes, which eases the creation of DCPS routes and enables the enforcement of QoS at end-to-end level. Finally, we propose a scalable Peer-to-Peer (P2P) solution for performing content discovery and data transfer. This solution is based on the REsource LOcation And Discovery (RELOAD) framework, one of the latest standards of the Internet Engineering Task Force (IETF). This solution provides rendezvous function for locating sensors, actuators, and applications that share a common data-space or topic

11 ii of interest using a DCPS approach. Once the rendezvous is complete, nodes use existing RELOAD features for exchanging information. We demonstrate the proposed solution scales well for large deployments and it is robust against node failures.

23 x List of Figures 3.9 DDS-IS performance for filtering and transforming data Average latency for different payloads and number of routes (topics) active Total average throughput for different payloads and number of routes (topics) active CPU impact for different payloads and number of routes (topics) active System topology used in 1 to N performance experiments Average latency for different payloads when demultiplexing data from 1 to N Throughput per subscriber for different payloads when demultiplexing data from 1 to N System topology used in M to N scalability experiments DDS traffic load on LAN C for both no-dds-is and DDS-IS scenarios Topology of the Health Service use-case DDS-IS are able to integrate incompatible information data-bases Current solutions provoke unnecessary and redundant traffic in the higher levels of the DDS-IS hierarchy Our solution allows for a better usage of the resources Node joining and content registering Session creation diagram Architecture layers Example of overlay prior to any DataRegister Example of overlay after several DataRegister A) Average latency for storing DCPS entities discovery information. B) Average latency for obtaining all the discovery information of a particular DCPS-Entity-Group Average latency for obtaining the discovery information for a particular DCPS-Entity-Group and start exchanging data with a node of that group

48 Chapter 1 Introduction Current uses of the Internet are mostly built on request-response exchanges between pairs of computers. This model has been very successful at providing shared remote access to resources and point-to-point communications for applications including , remote file and database access, web access, etc. Transmission Control Protocol (TCP), the most widely used Internet protocol, is also geared towards provided reliable point-to-point communications, to the extent that each TCP message embeds the location of the sending and receiving endpoints. The advent of ubiquitous Internet connectivity and cheap network-connected devices has greatly expanded the communication needs of applications using the Internet. For these new kinds of applications the request-response communications model is cumbersome and limited whereas other communication models, such as publish-subscribe [21], provide a more natural fit. The nature and historical context of this problem is well described in Van Jacobson et al. seminal work [41]: The engineering principles and architecture of today s Internet were created in the 1960s and 70s. The problem networking aimed to solve was resource sharing [...]. The communication model that resulted is a conversation between exactly two machines, one wishing to use the resource and one providing access to it. Thus IP packets contain two identifiers (addresses), one for the source and one for the destination host, and almost all the traffic on the Internet consists of (TCP) conversations between pairs of hosts. In the 50 years since the creation of packet networking, computers and their attachments have become cheap, ubiquitous commodities. The connectivity offered by the Internet and low storage costs enable access to a staggering amount of new content 500

49 18 Chapter 1. Introduction exabytes created in 2008 alone [24]. People value the Internet for what content it contains, but communication is still in terms of where. We see a number of issues that affect users arising from this incompatibility between models. Availability: Fast, reliable content access requires awkward, pre-planned, applicationspecific mechanisms like CDNs and P2P networks, and/or imposes excessive bandwidth costs. Security: Trust in content is easily misplaced, relying on untrustworthy location and connection information. Location-dependence: Mapping content to host locations complicates configuration as well as implementation of network services. The direct, unified way to solve these problems is to replace where with what. Host-to-host conversations are a networking abstraction chosen to fit the problems of the 60s. We argue that named data is a better abstraction for today s communication problems than named hosts. In contrasts to the request-response interaction model, in which a client sends a request and a server returns a response, this Thesis that in a sense is aligned with Jacobson s vision focuses on the publish-subscribe interaction model. In the publish-subscribe model information producers (publishers) are decoupled from information consumers (subscribers) via the publish-subscribe service. Publishers generate events or update a shared information space, subscribers express their interest in certain patterns of events or parts of the information space, and the publish-subscribe service delivers the desired information from publishers to subscribers. This vision is well aligned with recent projects aimed at defining a future publishsubscribe-based Internet that substitutes the current Internet design, in particular Publish-Subscribe Internet Routing Paradigm (PSIRP) [20] [22] and Publish Subscribe Internet Technology (PURSUIT) [82]. 1.1 Motivation In 2004, the Object Management Group (OMG) [73] adopted Data Distribution Service (DDS) [67], a specification that standardizes data-centric publish-subscribe communications. This specification defines a standardized Application Programming Interface (API) for exchanging information following a Data-Centric Publish- Subscribe (DCPS) model. In this model, publishers and subscribers exchange information through a distributed cache known as data-space (also known as DDS domain). Another relevant characteristic of DCPS is that exchanged information

50 1.1. Motivation 19 is not opaque to the middleware. In this way, DDS is able to provide advanced functionalities such as data-caching or content-based-filtering. The DDS standard is still growing. Proof of the activity around this standard is the fact that just in the last two years the OMG has published four of the six documents currently constituting the DDS standard family. As a consequence of the relative novelty of DDS, there are still several problems to solve. DDS was originally designed for isolated Local Area Networks (LANs) in which data sources (publishers) and data consumers (subscribers) are located in the same geographical location. Problems arise if a subscriber has interest in data being originated in a different data-space, especially if that data-space is separated by a bandwidth-constrained network, or if a Network Address Translation (NAT) or firewall service is present Domain interconnection Consider an example scenario with a company that relies on DDS for delivering some data-content internally. In addition, this company provides certain services that require to expose a subset of that data-content to another company. A simplistic solution to the previous use case would be to merge the two dataspaces. That is, to implement a service that publishes and announces all the topics from one data-space to the other. However, the direct merging of the data-spaces would create some significant problems. The first problem is related to scalability. Every publication available in one data-space will be indiscriminately announced in the other data-space. This may be wasteful in resources in situations where there are no consumers to that information on the second data-space. Other problems are concerned to data compatibility. For instance, each dataspace may have its own (possibly different) data model (i.e., data names, types, and/or definitions). These models could be incompatible even if they refer to the same data items. For example, temperature data might be expressed in Celsius in one data-space and in Fahrenheit in the other one. Therefore, in order to keep the end-to-end application logic unchanged, data should be automatically transformed by the publish-subscribe infrastructure when bridging data-spaces with different data models. As an additional benefit, this will ease the integration between new and legacy DDS applications. Access control and information confidentiality adds another dimension to this problem: Some parts of the data should be confined in a data-space whereas other

51 20 Chapter 1. Introduction data should be public. Another relevant issue is related to the possibility of having different delivery requirements for the information that flows between and within the different interconnected data-spaces. For example, system administrators should be able to establish the Quality of Service (QoS) requirements that domains must meet in order to be bridged. The current DDS specifications do not address any of these domain interconnection issues. Consequently, one aspect of this research focused on the design of a service that connects disjoint DDS data-spaces (i.e., a bridging service) while also solving the problems stated above Signaling DDS models information exchange using six different entities: DomainParticipant, Topic, Publisher, Subscriber, DataWriter (DW), and DataReader (DR). DomainParticipant represents the connection from an application to a shared dataspace. Topic represents a stream of information of interest, it has an associated topic name and data-type. Publisher is the object responsible for sending information. Subscriber is the object responsible for receiving information. DataWriter (DW) is the object the application uses to write data to a specific topic. Each DW is bound to a specific topic (and therefore data-type) and belongs to a publisher. DataReader (DR) is the object the application uses to receive data on a specific topic. Each DR is bound to a topic (and therefore data-type) and belongs to a subscriber. All these DDS entities have QoS attached to them specifying non-functional aspects of the information exchange. In order to exchange data-content, DDS entities must discover each other. DDS includes a discovery mechanism. DDS discovery is the process used by the publishsubscribe service to find the entities which share a particular topic and meet a set of QoS requirements.

52 1.2. Goals 21 Once a DW discovers a matching DR the corresponding applications can start exchanging data. Originally, the DDS specification standardized the programming model and API for developing DCPS applications. Despite of the fact that it defined the entities for exchanging discovery information, it did not specify the network protocol used by DDS implementations in a way that would allow different implementations to interoperate. The OMG addressed this issue by specifying the Real-time Publish- Subscribe Wire Protocol DDS Interoperability Wire Protocol (DDS-RTPS) [69]. The DDS-RTPS specification defines the so called DDS Simple Discovery Protocol (SDP) to perform the discovery. This discovery protocol was designed for relatively small environments (up to a few thousands nodes connected through a LAN). In this regard, the current DDS specifications do not address the problem of universal entity identification and universal data-content identification. Another existing limitation impacts scalability. Most of current DDS implementations rely on using multicast or a centralized server for discovery. In this sense, DDS-RTPS does not address the problem of entity or data-content discovery in large-scale deployments, nor provides any mechanism for performing NAT traversal. Fortunately, the DDS-RTPS specification allows DDS implementations to define their own discovery protocols, therefore is possible to define new protocols that address these issues. The introduction of a bridging service between DDS entities (as the one proposed in this Thesis) creates new problems. In particular, current DDS QoS policies were designed for single-domain-based scenarios, and they may behave unexpectedly in environments were a bridging service is present. In addition, applications need new mechanisms for negotiating what contents are propagated through different data-spaces. In short, the introduction of intermediate nodes in DDS creates the need of new mechanisms that support the negotiation of QoS policies among remote nodes, along with the creation of publications and subscriptions across DDS bridging services. In order to address these issues, this Thesis defines and proposes the use of a signaling protocol for DDS. 1.2 Goals The main objectives of the Thesis are organized in two categories: DDS data-space interconnection and session signaling. DDS data-space interconnection: Objective: To design a data-space bridging service for DDS.

53 22 Chapter 1. Introduction Objective: To increase the extensibility and flexibility of DDS systems, easing DDS systems evolution. Objective: To implement and evaluate the designed data-space bridging service for DDS. Objective: To study the impact of the proposed service in DDS QoS. Session signaling in DDS: Objective: To design an universal entity and data-content naming format for DCPS environments. Objective: To design a protocol for DCPS entities discovery and session establishment in DDS. Objective: To evaluate the scalability and robustness of the proposed solution. 1.3 Solution overview The solution proposed in this Thesis consists of three parts: 1. DDS data-space Interconnection Service (DDS-IS): We propose a solution for the interconnection of DDS data-spaces that is compliant with DDS specification. We experimentally demonstrate that it provides data transformation between data-spaces, performs content-based filtering, increases DDS scalability, and provides QoS adaptation between remote entities. 2. Data-Centric SIP (DCSIP): We propose the fundamentals and rationale of a protocol for session signaling in DCPS environments. This protocol solves DCPS entity and data-content discovery in DDS-IS-based deployments. It also allows the creation of sessions between nodes, which eases the creation of DCPS routes and the enforcement of QoS at end-to-end level. We also propose a mechanism for universal identification of DCPS entities and data-content. 3. DCPS usage for REsource LOcation And Discovery (RELOAD): We propose a Peer-to-Peer (P2P) solution for discovering and accessing to content using a DCPS approach. Our solution is based on the RELOAD framework, one of the latest standards of the Internet Engineering Task Force (IETF). Our solution is not limited to DDS-based scenarios, and it is valid even if only a few (or none) of the nodes in the overlay supports DDS. We experimentally demonstrate that our solution, based on a P2P architecture, features great scalability and robustness for deploying DCPS-based systems.

54 1.4. State of the art and related work State of the art and related work This section studies the state of the art that provides the context for the research carried out in this Thesis Publish-subscribe interaction model The key feature of publish-subscribe-based architectures is that information consumers (subscribers) and information producers (publishers) are decoupled in time, space, and synchronization [65] [21]. Several Message Oriented Middleware (MOM) [7] implementations, which are based on asynchronous message-passing, use the publish-subscribe model for exchanging data. For example, Java Messaging Service (JMS) [74] and Advanced Message Queuing Protocol (AMQP) [3] are MOM-based technologies that allow applications to deliver messages using a publish-subscribe approach. Other relevant approaches are also remarkable. For instance, [96] and [95] propose a publisher-subscriber architecture for distributed real-time systems based on CORBA middleware [68], and [107] proposes an XML-based solution for deploying publish-subscribe systems. Another interesting solution is SIENA [12], a publishsubscribe event notification service that takes advantage of a P2P architecture. We will study this and other P2P-based approaches in the next section. Directly related to our work, the authors of [75] [93] propose DDS, a middleware that provides data-centric publish-subscribe content distribution. DDS was adopted in 2004 by the OMG [73] [67] as the standard middleware for DCPS communications. Since the original DDS specification in 2004, more than eight different implementations of the standard have appeared [72]. The DDS standard is still growing, and several extensions to the original specification have been adopted over the last years. For example, DDS-RTPS [69] defines a protocol that allows different DDS implementations to interoperate. Another noteworthy example is the Extensible And Dynamic Topic Types For DDS (DDS-XTypes) [71] specification, which allows DDS applications to define new types in runtime. However, and due to the novelty of the standard, several problems, such as the interconnection of disjoint domains, remain unsolved.

55 24 Chapter 1. Introduction Scalability in publish-subscribe systems Publish-subscribe technologies are usually implemented as brokered services. This leads to well known issues that derive from centralized architectures such as single point of failure and bottlenecks. To cope with these issues, many solutions have been proposed in the literature. For instance, [23] proposes a hierarchical multi level (clusters and super-clusters) JMS compliant approach for interconnecting grids. It features scalability, loadbalancing, and fault-tolerant federation. However, this approach does not support data transformation, which is one of our initial objectives. Moreover, it does not take into account QoS related issues, making difficult its usage in critical scenarios. The authors of [12] propose SIENA, a publish-subscribe event notification service that utilizes two different architectures for distributing event notifications: hierarchical client-server and P2P. Regarding the routing scheme (i.e., the criteria that the system follows in each hop to decide how forward samples from publishers to subscribers), SIENA uses a filter-based approach (i.e., it makes routing decisions via successive content-based filtering at all nodes from source to destination). However, this work does not support multicast-based routing, which is one of the communication schemes that DDS supports. Moreover, this work does not consider data transformation nor QoS enforcement. From the authors of SIENA, [13] proposes a routing schema that combines broadcast and content-based filtering for improving the scalability of the system. As in the previous work, it does not consider data transformation nor QoS enforcement. Similarly, [11] proposes Kyra, a routing scheme based on content-based filtering that applies load balancing mechanisms. Again, this work does not support multicast-based routing, data transformation, nor QoS enforcement. All the above-mentioned work utilize a content-based filtering approach. There are also existing approaches that use a multicast-based routing scheme. For example, the authors of [15] propose HOMED, a P2P overlay for supporting distributed publish-subscribe systems. It constructs a flexible and efficient event dissemination tree by organizing participants based on their interest. Another interesting work is [98], which proposes an alternative for creating the multicast dissemination tree. In particular, it proposes to map subscriptions and events to rendezvous nodes; for that, it uses a combination of the domain schema identifier and the numbers of attributes in the subscriptions or the events. None of the above two solutions considers the problem of data transformation nor QoS enforcement. The authors of [104] propose to partition the event space among the peers in the system, and to broadcast events and subscriptions to all nodes in the system.

56 1.4. State of the art and related work 25 These broadcasts are attenuated by filters installed at peer nodes depending on the subscriptions stored in the system. However, this approach may require all peers in the system to be contacted to install a subscription. To reduce the number of peers to be contacted, [25] proposes Meghdoot, a content-based publish-subscribe system built on top of Content Addressable Network (CAN) infrastructure. Again, none of these two solutions considers the problem of data transformation nor QoS enforcement. More recently, the Apache QPiD project [4] includes a federation mechanism for AMQP [3] that increases the scalability of the system by establishing dedicated routes between brokers. Also related to AMQP, [60] provides a systematic method for configuring broker federation in AMQP-based systems. A different approach is included in [108], where the authors combine clustering, content-based routing, and document flooding for providing scalable publish-subscribe communications for MANETs. In all these cases, they deal only with scalability problem, leaving open issues such as data transformation. Related to DDS, the authors of [16] and [17] present REliable and VErsatile News delivery support for agencies (REVENGE). REVENGE is a middleware used to distribute breaking news among heterogeneous clients while guaranteeing QoS, availability, and reliability. This work has two main contributions: a self-organizing P2P routing substrate to facilitate reliable data delivery between mobile nodes, and a relay-based DDS support infrastructure with limited overhead to enable scalable, Internet-wide data dissemination. REVENGE relay nodes allow for the exchange of information among different DDS domains. However, this communication is bound to a specific topic/data-type, which limits the flexibility of the system. Finally, in [77] the authors propose an architecture to interconnect DDS dataspaces with Enterprise Service Bus (ESB), thus allowing to exchange information between the embedded world and the enterprise system. This work is mainly focused on increasing the interoperability of DDS by providing a routing substrate between ESB and DDS. However, it does not provide any mechanism to increase the scalability on the DDS system (like the data aggregation proposed in this Thesis), and does not provide mechanisms for performing DDS data transformations (except the strictly necessary for integrating ESB and DDS) Resource discovery and RELOAD One of the current issues of DDS is the scalability of its discovery protocol in largescale deployments (deployments with thousands of nodes). In the literature we can find several proposals that take advantage of P2P-based architectures for performing resource discovery. For example, [90] proposes Spi-

57 26 Chapter 1. Introduction der, a framework for Web Service discovery. It supports the lookup of Web Services based on keywords, ontologies, or behavior. Also related to Web Service discovery, the authors of [94] propose a scalable indexing system for P2P-based architectures that incorporates load-balancing mechanisms. However, these two works are focused in Web Service discovery, and therefore they are not suited to support the specific requirements of a publish-subscribe middleware (for example, DDS requires to treat publishers, subscribers, and participants differently). SOLAR [14] proposes the Context-Sensitive Resource Discovery (CSRD). This work, mainly focused on context-aware architectures, does not take into account QoS related issues, making it difficult to be used in more critical scenarios. Regarding discovery in DDS, the paper [92] addresses the problem of scalable discovery in DDS environments by applying Bloom filters. However, this work only focuses on the discovery problem itself, not addressing other issues such as maintaining an overlay to provide fault-tolerance, or NAT-traversal. In the time of writing, the IETF is in the process of approving RELOAD [44] as standard. RELOAD is an IETF protocol for building and maintaining P2P systems on the Internet. In particular, it provides mechanisms for resource discovery and for data-content storage and retrieval. A number of papers have aimed to demonstrate the feasibility of RELOAD as an stable and reliable P2P architecture. For instance, RELOAD performance in wireless environments has been studied in [58]. In [56], a study of the performance of RELOAD on an Interactive Connectivity Establishment (ICE)-based environment for NAT traversal is done. Another contribution is [28], which addresses the problem of resource discovery and notification in Constrained Application Protocol (CoAP) environments. Finally, and very close to our work, the authors of [2] propose a publish-subscribe protocol built atop of RELOAD. Although it provides the basic features needed for a publish-subscribe system, it relies completely on tunneling publish-subscribe samples, which hinders its flexibility and extensibility. Moreover, this work does not cover the discovery of specific DDS entities, such as participants DDS and SIP In order to provide DDS of a signaling protocol, we initially consider using Session Initiation Protocol (SIP). SIP [83] is an IETF [39] protocol specification for session signaling in client-server-based architectures. At the beginning of this Thesis, there were no other publications that integrated SIP and DDS. In [51] we proposed a basic DDS-SIP gateway to connect remote DDS

58 1.5. Main contributions 27 domains. This was the first publication that defined the concept of DDS session. This work defined DDS session as a logical channel that connects two remote DDS domains. However, we finally decided to use RELOAD a P2P approach instead of SIP a client-server approach because it provides greater scalability. We can find another example of SIP-DDS integration in [26]. This paper proposes a system that integrates SIP and DDS for supporting DDS-based applications over QoS-enabled IP WANs. It also defines an extension to Session Description Protocol (sdp) that supports the media description of DDS sessions. As a downside, this solution is limited by the client-server architecture SIP is bound to. 1.5 Main contributions The main contributions of the Thesis are: 1. The design, implementation, and performance evaluation of a DDS data-space Interconnection Service (DDS-IS). 2. A study of the impact of DDS-IS in QoS enforcement. 3. The definition of universal identifiers for DDS domains and topics. 4. The design of a DCPS-oriented protocol for session establishment, Data-Centric SIP (DCSIP). 5. The design, implementation, and evaluation of a new RELOAD usage for providing DCPS entity and data-content discovery. We definitively believe these contributions constitute an important step toward making DDS a mature and seamless technology, aligned with the envisaged namedbased Internet Published works Part of our work is already available to the research community through several publications in indexed journals and conference proceedings. In addition, as a result of the conducted work we have applied for a patent that is currently being pursued by Real-Time Innovations Inc. Contributions directly related to this Thesis:

60 1.6. Document structure J. Povedano-Molina, J. M. Lopez-Vega, J. Sanchez-Monedero, and J. M. Lopez- Soler, Instant messaging based interface for data distribution service, in XIII Jornadas de Tiempo Real, [79] 5. J. Povedano-Molina, J. M. Lopez-Vega, and J. M. Lopez-Soler, EMDS: an extensible multimedia distribution service, in Real-time and Embedded Systems Workshop, May [Online]. Available: SMCS/rt/pr/pdf/Povedano-MolinaLopez-VegaLopez-Soler_EMDS.pdf [78] 6. J. Sanchez-Monedero, J. Povedano-Molina, J. M. Lopez-Vega, and J. M. Lopez- Soler, Bloom filter based discovery protocol for DDS middleware, Journal of Parallel and Distributed Computing, vol. 71, no. 10, pp , May [Online]. Available: [92] 7. J. M. Lopez-Soler, J. M. Lopez-Vega, J. Povedano-Molina, and J. J. Ramos- Munoz, Performance evaluation of Publish/Subscribe middleware technologies for ATM (air traffic management) systems, in Workshop on Real-time, Embedded and Enterprise-Scale Time-Critical Systems, [49] 1.6 Document structure The remainder of the document is organized as follows. In Chapter 2 we study the technologies that provided the starting point of this Thesis. We have divided the proposed solution in three chapters that correspond to the division described in Section 1.3. Each one of these chapters includes its own introduction, motivating use cases, design, experiments, results, discussion of results, and conclusions. This structure eases the reading of the document, as each chapter is self-contained. In particular, Chapter 3 includes the design, implementation, and performance evaluation of a DDS data-space Interconnection Service; it also includes an study of the impact of this service in QoS enforcement. In Chapter 4 we present the design of a DCPS-oriented protocol for session establishment, named DCSIP. In Chapter 5 we present the design, implementation, and evaluation of a new usage for RELOAD that provides DCPS entity and data-content discovery. Finally, in Chapter 6 we discuss related work, and in Chapter 7 we summarize the main conclusions of our work.

61

62 Chapter 2 Background In this chapter we study the technologies that provided the starting point of this Thesis. First, we study the publish-subscribe interaction model, which is an alternative to the request-response model. In particular, we study the DDS specification, which is a standardized publish-subscribe middleware for data dissemination on real-time distributed systems. Regarding signaling and resource discovery, we study SIP, an application protocol for session signaling, and RELOAD, a standardized P2P-based framework for highly scalable resource discovery. Finally, we introduce CoAP, a HTTP-inspired protocol for exchange data-content between highly constrained devices. 2.1 Publish-subscribe interaction model Over the last few years, publish-subscribe communication systems are gaining momentum and popularity, as verified beyond doubt by the existence of successful projects like PSIRP [20] [22], and PURSUIT [82], which is a continuation of PSIRP. These projects are aimed to define a future publish-subscribe-based Internet that substitutes the current Internet design, based on TCP/IP [101]. In contrasts to the request-response interaction model, in which a client sends a request and a server returns a response, in the publish-subscribe interaction model

63 32 Chapter 2. Background producer producer request consumer response consumer consumer publisher publisher Publish-subscribe service subscribe subscriber subscriber subscriber delivery Figure 2.1: Request-response and publish-subscribe interaction models. information producers (publishers) are decoupled from information consumers (subscribers) via the publish-subscribe service. Publishers generate events or update a shared information space, subscribers express their interest in certain patterns of events or parts of the information space, and the publish-subscribe service delivers the desired information from publishers to subscribers. (see Figure 2.1 [48]). The key feature of publish-subscribe-based architectures is that information consumers (subscribers) and information producers (publishers) are triply decoupled [21]. In particular, publish-subscribe applications are decoupled in space (publishers and subscribers can be located anywhere), in time (there is no need of simultaneous end-point availability), and in synchronization/flow (publishers are not blocked while producing events, and subscribers can get notified about the occurrence of some event while performing some concurrent activity). We can classify publish-subscribe systems according to the mechanism that they use for specifying what are the events of interest [21] (i.e., the scheme). The most widely used schemes are topic-based, content-based, and type-based. In topicbased schemes, events are classified according to a label, the topic. In content-based schemes, events are classified using the content of the events. Finally, in type-based schemes, events are distinguished according to their structure. Several MOM [7] implementations, which are based on asynchronous messagepassing, use the publish-subscribe model for exchanging data. For example, JMS [74] and AMQP [3] are MOM-based technologies that allow applications to deliver messages using a publish-subscribe approach. Other relevant approaches are also remarkable. For instance, [96] and [95] propose a publisher-subscriber architecture for distributed real-time systems based on CORBA middleware [68], and [107] proposes an XML-based solution for deploying publish-subscribe systems. Another relevant work is SIENA [12], a publishsubscribe event notification service which was studied in Section Directly related to our work, the authors of [75] [93] propose DDS, a middleware that provides data-centric publish-subscribe content distribution. In the next sec-

64 2.2. Data Distribution Service (DDS) 33 tion we study OMG DDS, which is the standard specification for this middleware. 2.2 Data Distribution Service (DDS) DDS [67] is a specification adopted by the OMG [73]. DDS standardizes DCPS communications in distributed scenarios. For more information about the DCPS model, please refer to Section Since the original DDS specification in 2004, several implementations of the standard have appeared [72]: RTI Connext [89], a COTS implementation of DDS which supports multiple languages and platforms; OpenSplice [81], a partially open implementation of DDS which also supports multiple languages and platforms; OpenDDS [62], an open source C++ implementation; MilSOFT DDS [61], a COTS implementation of DDS; OCERA ORTE [63], an open source implementation of RTPS 1.0.; CoreDX [105], a COTS implementation of the DDS specification; Inter- COM DDS [47], an implementation of the DDS minimum profile; MicroDDS [30], a DDS implementation for small devices, micro-controllers and embedded systems; and BEE-DDS [100], a Java implementation of the OMG DDS standard. Over the last few years, DDS has been deployed in many contexts ranging from mission critical systems to financial environments [85] [86] [19] [87] [80]. All these scenarios have in common that different collocated entities exchange data samples coming from different sources (for example, position sensors, stocks, etc.) in a well controlled environment (i.e., a data-space). In addition, this information is exchanged with certain guarantees such as reliability or deterministic time-delivery. DDS uses a data-centric publish-subscribe model along with a rich set of QoS profiles that can be tuned to fit the communication demands of each specific scenario. DDS specification defines two different interface levels [75] and an interoperability layer [69]: The Real-time Publish-Subscribe Wire Protocol DDS Interoperability Wire Protocol (DDS-RTPS) interoperability layer. This layer is based on the Real Time Publish Subscribe (RTPS) protocol, and allows different DDS implementations to interoperate. This layer defines the discovery protocol, data representation format, and message format DDS applications must use in order to interoperate. The mandatory Data-Centric Publish-Subscribe (DCPS) interface level. This defines a programming model and API for developing applications using a data-centric publish-subscribe approach.

65 34 Chapter 2. Background The optional Data Local Reconstruction Layer (DLRL) interface level. This layer acts as an interface between the application layer and the DCPS layer. DLRL is object oriented and was specified to ease the integration of DCPS in user applications. This level allows developers to access to DDS data-content through object attributes. In what follows under this section, we focus on studying the DCPS layer and the DDS-RTPS protocol, which have a key role in our research Data Centric Publish Subscribe layer (DCPS) The DCPS layer specification [67] defines the Data-Centric Publish-Subscribe (DCPS) programming model and its associated API for developing distributed applications. DCPS programming model is based in two main concepts: datacentricity and publish-subscribe. Data-centric oriented middleware, such as DDS, contrasts with MOM in that message content is not opaque to the middleware. Using the functionality provided by the DCPS layer, DDS is able to interpret exchanged data-samples. In this way, DDS is able to provide advanced functionalities such as data-caching or contentbased-filtering. The DCPS programming and communication model is based on the publishsubscribe paradigm, and therefore DCPS-based applications are decoupled in space, time, and synchronization. In addition, DCPS-based applications are also decoupled in platform: DCPS developers can deploy distributed systems using diverse operating systems, hardware architectures, and languages. The DCPS layer defines the following concepts: Data-space: Also known as domain, it is an aggregation of data of different datatypes and sources. More precisely, a data-space is a distributed cache, whose content is accessed and updated by subscribers and publishers. Topic: A portion of a data-space that groups data of the same data-type. Although we could think that DDS uses a topic-based scheme, this is not strictly true, since topics are usually associated to a type, which is typical of type-based schemes. Moreover, DDS supports the use of content-based filtering, which is a typical characteristic of content-based schemes. Publisher: An entity that pushes data-content updates to one or more topics. Subscriber: An entity that is interested in data-content updates from one or

66 2.2. Data Distribution Service (DDS) 35 Application Application Application DomainParticipant DomainParticipant DomainParticipant Publisher Subscriber Publisher Subscriber DW DR DW DW DR DR Reliable transport QoS data-space (domain) Topic A Topic B Best-effort transport QoS Figure 2.2: DDS concepts and entities. more topics. According to DCPS, publishers and subscribers exchange data-content within a particular shared data-space. DDS applications access and update the data-content in this data-space using topic-specific endpoint entities, respectively called DRs and DWs. Applications access to all these entities through DomainParticipants, which are the interface between applications and DDS data-spaces. Under the DCPS model, when a publisher wants to update data-content belonging to certain topic, the middleware assumes responsibility for managing its delivery to the proper interested peers (i.e., to the associated subscribers). As a result, since DDS relieves the application from burden of transmitting and managing the data, the application logic is drastically simplified. Figure 2.2 illustrates main DDS concepts and relationships. DDS states that within a data-space data objects are completely identified by a topic name, a data-type, and optionally by a key. This key is useful to identify different instances of the same topic. In order to exchange data-content, DWs and DRs (referred to as endpoints) have to find compatible endpoints. The process of finding compatible entities is known as discovery. DDS performs discovery using a set of built-in publications (i.e., a set of mandatory-to-implement publications), which are handled using the so-called Built-in DataReaders (B-DRs) and Built-in DataWriters (B-DWs) (see Figure 2.3). The discovery function allows DDS applications to automatically discover available (and compatible) publications and subscriptions. Another feature of DCPS is the support of QoS policies. A QoS policy repre-

67 36 Chapter 2. Background DomainParticipant Participant Discovery Phase Advertises this participant Discovers other participants Builtin DataWriter Builtin DataReader Participant DATA DCPSParticipant builtin topic Participant DATA DCPSParticipant builtin topic Endpoint (Reader/ Writer) Discovery Phase Advertises this Participant's DataWriters and DataReaders Discovers other Participant's DataWriters and DataReaders Builtin DataWriter Builtin DataReader Builtin DataWriter Builtin DataReader Publication DATA DCPSPublication builtin topic Subscription DATA DCPSSubscription builtin topic Publication DATA DCPSPublication builtin topic Subscription DATA DCPSSubscription builtin topic Network Figure 2.3: DCPS built-in entities for discovery purposes. sents an agreement among a group of DDS entities. This agreement determines the behavior of the middleware, and releases the application logic of certain tasks, such as communication reliability. In case of QoS infringement, DDS detects it and optionally triggers proper actions. Each DDS entity has its own QoS policies and data-caching attributes. As an example, a DW and a DR can negotiate the level of reliability of the communications (best effort or reliable delivery) by using the RE- LIABILITY QoS. It is also possible to guarantee a maximum period between data updates (DEADLINE QoS), or to filter samples according to their content or timing Real-time Publish-Subscribe Wire Protocol DDS Interoperability Wire Protocol (DDS-RTPS) The original DDS specification did not enforce any particular lower level implementation requirement or recommendation, such as the use of a particular underlying transport protocol. For the sake of vendors interoperability, the OMG defined the so-called Real-Time Publish-Subscribe Wire Protocol Real-time Publish-Subscribe Wire Protocol DDS Interoperability Wire Protocol (DDS-RTPS) [69]. DDS-RTPS is based on Real-Time Publish-Subscribe (RTPS), a protocol originally approved by the International Electrotechnical Commission (IEC) [32] as part of the Real-Time Industrial Ethernet Suite IEC-PAS [31]. After applying some modifications, the OMG adopted RTPS as the standard wire protocol for DDS. DDS-RTPS specifies the data representation, message format, and discovery proto-

68 2.2. Data Distribution Service (DDS) 37 col that DDS implementations must use in order to interoperate. In order to exchange data-content, DDS entities must discover each other. DDS includes a discovery mechanism. DDS discovery is the process used by the publishsubscribe service to find the entities which share a particular topic and meet a set of QoS requirements. One of the goals of DDS-RTPS is to allow entities belonging to different DDS implementations to perform discovery. DDS-RTPS discovery reutilizes most of the existing DDS core entities. Specifically, the correspondence between DDS-RTPS and DDS entities is shown in Table 2.1. Table 2.1: Correspondence DDS and DDS-RTPS entities. DDS entity DomainParticipant DataWriter DataReader Built-in DataWriter Built-in DataReader DDS-RTPS entity Participant Writer Reader ENTITYID SPDP BUILTIN * WRITER ENTITYID SPDP BUILTIN * READER According to DDS-RTPS, any discovery protocol must include the following phases: Participant Discovery Protocol (PDP) and Endpoint Discovery Protocol (EDP). The purpose of PDP is to discover new participants in the data-space. Whenever a new participant is discovered, the EDP procedure is triggered to exchange local and remote endpoints information between two participants. Different implementations may choose to support multiple PDPs and EDPs, possibly vendor-specific. As long as two participants have at least one PDP and EDP in common, they can exchange the required discovery information. For the sake of interoperability, every DDS-RTPS implementation must support at least the SDP. This protocol consist of the Simple Participant Discovery Protocol (SPDP) as PDP protocol, and the Simple Endpoint Discovery Protocol (SEDP) as EDP protocol. Figure 2.4 depicts the typical SDP sequence diagram. It consists of the following phases: Bootstrapping: SDP discovery starts using a list of known hosts. This list contains the locators (typically unicast or multicast Internet Protocol (IP) addresses) for which a participant will announce its presence. Alternatively, if there are no specified IP addresses, default addresses will be used. Both options can be used together.

70 2.2. Data Distribution Service (DDS) 39 Node A Node B Participant A whish to inform about its Endpoints Participant B whish to mach A2, A4 and F We omit PDP information The HB is the same for both Participants participant A DATA participant B DATA Participant DATA heartbeat / BF Refresh period participant A Bloom filter Network messages SDP SDPBloom filter SDPBloom matched Endpoints DataWriter A1 created DataWriter A2 created DataWriter A3 created... DataWriter An created B can not match any topic Participant DATA heartbeat / BF Refresh period Node B's storage participant A Bloom filter Subscribe A2 Subscribe A4 Subscribe F B creates A2, A4 and F subscriptions B can match A1-N topics, topic F is a false positive B matchs A2, A4, F and tries to subscribe A2,A4,F SDP SDPBloom I have not F participant A BF Participant DATA heartbeat / BF Refresh period B deduces it was a false possitive and annotates it Figure 2.5: SDP-Bloom sequence diagram. supports SEDP, they exchange endpoint information. Endpoint information consists of a topic name, a topic type, and a set of QoS parameters. Using this information, each participant checks if local and remote endpoints are compatible, and if so, they start a publish-subscribe communication SDP Bloom In this section we introduce SDP-Bloom, an alternative SDP that helps to understand the flexibility and potential of the DDS-RTPS specification. Due to the pure distributed nature of SDP, each participant stores information about discovered participants, associated publications, and subscriptions in a local database. According to SEDP, each participant sends the description of its associated endpoints to every discovered remote participant. Therefore, each participant receives all the discovered participants endpoint information, which can generate a high number of messages. In this regard, in [92] we proposed a new discovery protocol for DDS, the SDP- Bloom. Figure 2.5 depicts the typical SDP-Bloom discovery sequence diagram. During PDP phase, participants using SDP-Bloom send a Bloom-filter summarizing

71 40 Chapter 2. Background their associated endpoint information. This allows participants to avoid performing the EDP discovery phase when the received remote participant s Bloom filter does not match with any of the local endpoints. In this way, SDP-Bloom takes advantage of using Bloom-filters for reducing the network overhead in terms of number of messages and total traffic, at the cost of a small false positive endpoint matching rate. For a more in depth study of SDP- Bloom functionality and its impact on DDS performance, please refer to [92] Extensible Topic Types As mentioned before, the DDS specification states that a topic has an associated data-type. However, binding a data-type with a publication: hinders system evolution and backward compatibility makes impossible to use a priori unknown data-types prevents from maintaining different data-types views The OMG has recently standardized the DDS-XTypes specification [71]. This specification, among other features, relaxes the requirement of using compilation time static topic data-types. The use of static topic data-types is simple, efficient, and provides compile-time type-safety. However, it requires types to be known a priori and compiled into the code. This characteristic prevents the implementation of generic bridging services for DDS, such as the one proposed in Chapter 3. In this regard, the main functionality of a bridging service is to subscribe to certain data-content in a data-space and to publish the received data-content to a different data-space. The use of static topic data-types requires developers to declare the data-types that the bridging service will use for each data-space. Consequently, it is not possible to implement a generic DDS bridging service (i.e., one that accepts topics of any data-type) by using static topic data-types. In DDS, type schemas that is, the names and definitions of a type and its fields are represented by TypeObjects. A TypeObject consists of a TypeObjectKind and a list of members. In addition to using locally serialized TypeObjects, the DDS-XTypes specification states that a serialized form of these (called DataRepresentation) must also be published (and therefore can be received) automatically during DDS discovery.

72 2.3. Session Initiation Protocol (SIP) 41 This mechanism allows applications to dynamically discover topic data-types. Once the types have been discovered, participants can use the DDS Dynamic Topic Types API (also defined in the DDS-XTypes specification) to publish and to subscribe to topics of types unknown at compilation time. Since the OMG has recently published the final version of the specification, some of the results presented in this Thesis (in particular, the ones presented in Chapter 3) are based on the current (and preliminary) version of the specification. 2.3 Session Initiation Protocol (SIP) SIP [83] is an IETF [39] protocol specification. SIP was originally specified within the Multiparty MUltimedia SessIon Control working group (MMUSIC WG) [33], and later within the Session Initiation Protocol working group (SIP WG) [34] and the Session Initiation Protocol Project INvestiGation working group (SIPPING WG) [35]. Currently, the Session Initiation Protocol Core working group (SIPCore WG) [37] is maintaining and continuing the development of the core SIP specifications. SIP is an application-layer signaling protocol for creating, modifying, and terminating sessions with one or more participants. These sessions usually include VoIP calls, multimedia streaming, and multimedia conferences. SIP also provides a registration function that allows users to upload their current location. In this section we study the fundamentals of SIP, which is one of the protocols our work is based on. In this regard, in Chapter 4 we propose a protocol for session signaling in data-centric publish-subscribe environments; and in Chapter 5 we propose an extension to RELOAD, which is a specification originally [44] conceived as an evolution of SIP Overview of SIP functionality SIP is an application-layer signaling protocol that can establish, modify, and terminate multimedia sessions (conferences) such as Internet telephony calls or videoconferences, among others. In this regard, SIP supports sdp, a protocol for describing the details of the session, such as the type of media, codec, or sampling rate. SIP dynamically manages sessions such as multicast conferences that is, it allows applications to invite participants to already existing sessions. In addition, it allows to negotiate and add (or remove) new media and formats to an existing session. SIP transparently supports name mapping and redirection services, which en-

73 42 Chapter 2. Background ables personal mobility. The SIP architecture originaly designed with a clientserver model also defines application servers to make the session highly configurable. SIP supports five facets of establishing and terminating multimedia communications: User location: determination of the end system to be used for communication. User availability: determination of the willingness of the called party to engage in communications. User capabilities: determination of the media and media parameters to be used. Session setup: ringing, establishment of session parameters at both called and calling party. Session management: including transfer and termination of sessions, modifying session parameters, and invoking services. From the previous facets, in next sections we focus on user location, session setup, and session management; which are the facets this Thesis is focused on Basic concepts of SIP The following terms have special significance for SIP [83]: Address-of-Record: A SIP or SIPS Uniform Resource Identifier (URI) that points to a domain with a location service. It can map the URI to another URI where the user might be available. Typically, the location service is populated through registrations. An address-of-record is frequently thought of as the public address of the user. Message: Data sent between SIP elements as part of the protocol. SIP messages are either requests or responses. Method: The method is the primary function that a request is meant to invoke on a server. The method is carried in the request message itself. INVITE and BYE are SIP method examples. Request: A SIP message sent from a client to a server, for the purpose of invoking a particular operation.

74 2.3. Session Initiation Protocol (SIP) 43 Response: A SIP message sent from a server to a client, for indicating the status of a request sent from the client to the server. Session: From the sdp specification [40]: A multimedia session is a set of multimedia senders and receivers and the data streams flowing from senders to receivers. A multimedia conference is an example of a multimedia session. SIP URI and SIPS URI: A SIP URI is a URI that identifies a communications resource. SIPS URIs use the same syntax than SIP URIs, but they specify that the resource must be contacted securely using Transport Layer Security (TLS) [6]. User Agent Client (UAC): A logical entity that creates a request, and then uses the client transaction state machinery to send it. The role of UAC lasts only for the duration of that transaction. In other words, if a piece of software initiates a request, it acts as a UAC for the duration of that transaction. If it receives a request later, it assumes the role of a User Agent Server (UAS) for the processing of that transaction. User Agent Server (UAS): A logical entity that generates a response to a SIP request. The response accepts, rejects, or redirects the request. This role lasts only for the duration of that transaction. In other words, if a piece of software responds to a request, it acts as a UAS for the duration of that transaction. If it generates a request later, it assumes the role of a UAC for the processing of that transaction. User Agent (UA): A logical entity that can act as both a UAC and UAS User location SIP offers a discovery capability [83]. In this regard, whenever a user wants to initiate a multimedia session, SIP discovers the current host(s) at which the target user is reachable. This is similar to the DDS discovery previously studied, where DDS entities (instead of users) discover compatible entities for exchanging DDS datasamples. In this regard, DDS application developers can take advantage of SIP location service for performing DDS discovery as we will see in Chapter 6. According to the Request For Comments (RFC) [83], the discovery process is frequently accomplished by SIP network elements, such as proxy servers and redirect servers, which are responsible for receiving a request, determining where to send it based on knowledge of the location of the user and then sending it there. To do this, SIP network elements consult an abstract service known as location service, which provides address bindings for a particular domain. These address

75 44 Chapter 2. Background bindings map an incoming address-of-record (e.g., to one or more URIs that are somehow closer to the desired user (e.g., biloxi.com). Ultimately, a proxy will consult a location service that maps a received URI to the UA(s) at which the desired recipient is currently residing. Registration creates bindings in a location service for a particular domain. A registration associates an address-of-record URI with one or more contact addresses. Thus, when a proxy for that domain receives a request whose Request-URI matches the address-of-record, the proxy will forward the request to the contact addresses registered to that address-of-record. There are many ways for establishing the contents of the location service. One way is administratively. In the above example, an existing corporate database is the source of the information about Bob. However, SIP provides a mechanism for a UA to create a binding explicitly. This mechanism is referred to as registration. Registration entails sending a REGISTER request to a special type of UAS known as a registrar. A registrar acts as the front end to the location service for a domain, reading and writing mappings based on the contents of REGISTER requests. The location service is then typically consulted by a proxy server that is responsible for routing requests for that domain. Figure 2.6 illustrates the overall registration process, where a user named carol registers her location. Note that the registrar and proxy server are logical roles that can be played by a single device in a network; for the sake of clarity the two are separated in this illustration. Also note that UAs may send requests through a proxy server in order to reach a registrar if the two are separate elements. In the figure we can also observe how the proxy sip.chicago.com requests for carol s location upon bob s INVITE request. SIP does not mandate a particular mechanism for implementing the location service. The only requirements are that a registrar for some domain must be able to read and write data to the location service, and a proxy or redirect server for that domain must be capable of reading those data. A registrar may be co-located with a particular SIP proxy server for the same domain. The location service is just an abstract concept. It generally contains information that allows a proxy to input a URI and receive a set of zero or more URIs. This information indicates the proxy where to send the request. Registrations are one possible way to create this location information, but not the only one. Alternatively, the administrator can also configure arbitrary mapping functions at his discretion.

76 2.3. Session Initiation Protocol (SIP) 45 bob UA )INVITE chicago.com V )Store Location 4)Query Registrar =======> Service <======= Proxy =======> A 5)Resp sip.chicago.com 1)REGISTER UA < cube2214a 6)INVITE carol Figure 2.6: SIP registration and invitation example. In addition, SIP registrations are not limited to only one device per user. For instance, a single user can register to the same URI both his SIP phone at home and the one in the office Session setup and management According to the RFC [83], whenever a UAC desires to initiate a session (for example, audio, video, or a game) it formulates an INVITE request (see Figure 2.7). The INVITE request asks a server for establishing a session. This request may be forwarded by proxies, eventually arriving at one or more UAS that can potentially accept the invitation. If the UAS accepts the session, it will send a 2xx response (which means success) back to the UAC. If the the invitation is not accepted, the UAS will send a 3xx, 4xx, 5xx or 6xx response (which respectively mean redirection, client error, server error,

77 46 Chapter 2. Background atlanta.com... biloxi.com. proxy proxy... Alice s Bob s softphone SIP Phone INVITE F > INVITE F2 100 Trying F > INVITE F4 < Trying F > < Ringing F6 180 Ringing F7 < Ringing F8 < OK F9 < OK F10 < OK F11 < < ACK F > Media Session <================================================> BYE F13 < OK F > Figure 2.7: SIP invitation dialog. and global failure). For more information about SIP responses, please refer to the RFC [83]. After possibly receiving one or more provisional responses, the UAC will get one or more 2xx responses or one non-2xx final response. The the UAC sends an ACK for every final response it receives. A 2xx response to an INVITE establishes a session. It also creates a dialog between the UA that issued the INVITE and the UA that generated the 2xx response. Therefore, when a UA receives multiple 2xx responses from different remote UAs (because the INVITE forked), each one of these 2xx responses establishes a different dialog. However, all these dialogs are part of the same call. A UA that supports INVITE must also support ACK, CANCEL, and BYE; as they are closely related to the INVITE request.

78 2.4. REsource LOcation And Discovery (RELOAD) 47 ACK confirms the reception of a 2xx response. CANCEL requests for the cancellation of an INVITE request. BYE requests for the finalization of an active session. SIP also provides mechanisms for modifying an existing session. The modification can involve changing addresses or ports, adding a media stream, deleting a media stream, and so on. This is accomplished by sending a new INVITE request within the same dialog that established the session. An INVITE request sent within an existing dialog is known as a re-invite. 2.4 REsource LOcation And Discovery (RELOAD) RELOAD [44] is an IETF protocol for building and maintaining P2P systems on the Internet. It provides a generic, self-organizing overlay network service, allowing nodes to efficiently route messages to other nodes and to efficiently store and retrieve data in the overlay. The working group responsible of the RELOAD specification is the Peer-to-Peer Session Initiation Protocol working group (P2PSIP WG) [36]. The original name of RELOAD was Peer-to-Peer Session Initiation Protocol (P2P-SIP). P2P-SIP was initially conceived as an evolution of SIP that replaced the relatively fixed hierarchy of SIP servers with a P2P overlay network [9]. The main objective of P2P-SIP was to increase the scalability of SIP by taking advantage of using a P2P approach instead of a client-server one. In this regard, the authors of [66] propose a mechanism for eliminating the need for the sign-up process and the servers for this purpose. RELOAD s original usage was focused on providing user registration, call setup, and instant messaging [99] [8]. In subsequent versions of the specification RELOAD was generalized in order to provide a P2P-based resource location and discovery service to any application (instead of only supporting the ones based on SIP) Main features RELOAD provides several features that are critical for a successful P2P protocol for the Internet [44] : - Security framework: P2P networks are often established among untrusted peers. RELOAD leverages a central enrollment server to provide credentials for each peer which can then be used to authenticate each operation. These credentials consist of a certificate assigned to each node that joins the overlay.

79 48 Chapter 2. Background - NAT Traversal: RELOAD is designed to function in environments where many (if not most) of the nodes are behind NATs or firewalls. RELOAD uses ICE [84] for providing NAT traversal functionality. - High Performance Routing: RELOAD has been defined with a simple, lightweight forwarding header, thus minimizing the amount of effort required by intermediate peers. - Pluggable Overlay Algorithms: For the sake of extensibility, RELOAD defines an abstract interface to the overlay layer to simplify implementing a variety of structured and unstructured overlay algorithms. Additionally, RELOAD uses a Chord-based Distributed Hash Table (DHT) [102] [103] as its default overlay algorithm. - Usage Model: RELOAD is designed to support a variety of (existent and future) applications. RELOAD allows the definition of new application usages, each of which can define its own data types, along with the rules for their use. In RELOAD, a Kind is the definition of a data type and its accessing rules. Therefore, new usages must define the Kinds they store in the RELOAD overlay. These properties allow RELOAD to adapt to different scenarios, while providing efficient and secure resource discovery Architecture and types of nodes A RELOAD overlay instance consists of a set of nodes arranged in a partly connected graph. Each node in the overlay is assigned a numeric Node-ID for the lifetime of the node which determines together with the specific overlay algorithm in use its position in the graph and the set of nodes it connects to. The Node-ID is also tightly coupled to a certificate. Figure 2.8 shows a trivial example of RELOAD overlay included in [44]. RELOAD defines two different types of nodes: - Peer: A host that is participating in the overlay. Peers are responsible for holding some portion of the data that has been stored in the overlay and also route messages on behalf of other hosts as required by the overlay algorithm. - Client: A host that is able to store data in and retrieve data from the overlay but does not perform routing or data storage for the overlay.

80 2.4. REsource LOcation And Discovery (RELOAD) Node Node Node Node Node Node Node Node Node Node 85 (Client) Figure 2.8: Trivial example of RELOAD overlay Operations As a P2P protocol, RELOAD defines a set of operations for joining, leaving, and maintaining the overlay. There are also operations for storing and retrieving data. In addition, RELOAD defines two new operations for interacting with other overlay nodes: Attach and AppAttach. Attach allows nodes to establish a direct TCP or User Datagram Protocol (UDP) connection to other overlay nodes. This connection is a direct channel between two Nodes exchanging RELOAD messages. AppAttach is similar to Attach, but in this case nodes use the generated connection for exchanging application data (and not RELOAD traffic). In this sense, one of the parameters of AppAttach is the Application-ID, which identifies connection s target application. RELOAD nodes send both Attach and AppAttach requests using the destination Node-ID (and not the IP address). In this sense, RELOAD allows nodes for establishing direct connections to any member of the overlay. This is feasible even if the target node is behind a NAT.

81 50 Chapter 2. Background Resource-ID Kind 1 Kind Value Value Value Value Value Data storage Figure 2.9: Example of data stored in RELOAD. RELOAD references the location of the data stored in the overlay by using an identifier which is known as Resource-ID, and its format depends on the particular DHT algorithm the overlay uses. Each location in the overlay may contain data elements corresponding to multiple Kinds. In addition, a resource location may contain multiple elements of a given Kind. Each Kind is identified by a Kind-ID, which is a code point either assigned by Internet Assigned Numbers Authority (IANA) or allocated out of a private range. Figure 2.9 depicts an example of resource location [44]. The RELOAD specification defines the following data models for storing data, though developers can define their own structures as part of a new RELOAD usage: single value: There can be at most one item in the set and any value overwrites the previous item. array: Many values can be stored and addressed by a numeric index. dictionary: The values stored are indexed by a key. Often this key is one of the values from the certificate of the peer sending the Store request.

82 2.5. Constrained Application Protocol (CoAP) Usages RELOAD s original usage was focused in providing a rendezvous service for SIP. In this way, communicating peers use RELOAD for establishing a connection, and then they perform the usual SIP negotiation [43]. There are other RELOAD usages like [42], which allows federating different SIP domains; or [27], which enables for managing the RELOAD overlay using Simple Network Management Protocol (SNMP). However, there is still no usage for supporting DDS. 2.5 Constrained Application Protocol (CoAP) CoAP [97] is an ongoing IETF protocol specification. The working group responsible of its specification is the Constrained RESTful Environments working group (CoRE WG) [38]. According to the current IETF draft [97], CoAP is a specialized web transfer protocol for use with constrained nodes and constrained (i.e., low-power, lossy) networks. CoAP provides a request-response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types. CoAP can be used not only between nodes on the same constrained network, but also between constrained nodes and nodes on the Internet. The latter is possible since CoAP can be translated to HyperText Transfer Protocol (HTTP) for integration with the web. Application areas of CoAP include different forms of Machine-to-Machine (M2M) communication, such as home automation, construction, health care or transportation. Areas with heavy use of sensor and actuator devices that monitor and interact with the surrounding environment. Although CoAP is conceived for using a request-response interaction model, [29] specifies a simple protocol extension for CoAP that enables CoAP clients to subscribe to certain resources. Once a client subscribes to a particular resource, the corresponding server will send notifications upon changes on the subscribed resource.

83

84 Chapter 3 DDS data-space Interconnection Service (DDS-IS) In this chapter, we consider the problem of communicating disjoint data-spaces that may use different schemas to refer to similar information. In this regard, we propose a DDS interconnection service capable of bridging DDS domains as well as adapting between different data schemas. These features constitute a first step for deploying DDS in large-scale deployments. A key benefit of our approach is that is compliant with the latest OMG specifications, thus the proposed service does not require any modifications to DDS applications. The chapter identifies the requirements for DDS data-spaces interconnection, presents an architecture that responds to those requirements, and concludes with experimental results gathered on our prototype implementation. 3.1 Motivation DDS was originally designed for isolated LANs in which data sources (publishers) and data consumers (subscribers) are located in the same geographical location. Problems arise if a subscriber has interest in data being originated in a different data-space, especially if that data-space is separated by a bandwidth-constrained network. This is the case of a company that relies on DDS for delivering some data-

85 54 Chapter 3. DDS data-space Interconnection Service (DDS-IS) content internally, and needs to share a subset of that data-content with another company. A simplistic solution to the previous use case would be to merge the two dataspaces. That is, to implement a service that publishes and announces all the topics from one data-space to the other. However, the direct merging of the data-spaces would create some significant problems. The first problem is related to scalability. Every publication available in one data-space will be indiscriminately announced in the other data-space. This may be wasteful in resources in situations where there are no consumers to that information on the second data-space. Other problems are concerned to data compatibility. For instance, each dataspace may have its own (possibly different) data model (i.e., data names, types, and/or definitions). These models could be incompatible even if they refer to the same data items. For example, temperature data might be expressed in Celsius in one data-space and in Fahrenheit in the other one. Therefore, in order to keep the end-to-end application logic unchanged, data should be automatically transformed by the publish-subscribe infrastructure when bridging data-spaces with different data models. As an additional benefit, this will ease the integration between new and legacy DDS applications. Access control and information confidentiality adds another dimension to this problem: Some parts of the data should be confined in a data-space whereas other data should be public. Finally, another relevant issue is related to the possibility of having different delivery requirements for the information that flows between and within the different interconnected data-spaces. For example, system administrators should be able to establish the QoS requirements that domains must meet in order to be bridged. In this chapter we propose a novel solution for inter-domain entity and data-type signaling, and data-content exchange in DDS environments. Our solution, named DDS-IS, is fully compatible with the current DDS standard specification and allows for advanced content-centric operations, like content-based filtering and data transformations. DDS-IS establishes a content-aware interconnection service between different data-spaces with the following distinctive features: a) Scalability. The overall traffic load is drastically reduced in comparison with direct data-space merging; b) Data model compatibility and confidentiality. The seamless communication between data-spaces with dissimilar data models (topics of different names or data-

86 3.2. Motivating use case example 55 types) was not possible so far. DDS-IS solves this issue by properly transforming and/or isolating topic samples, if necessary. c) Delivery guarantees. DDS-IS establishes QoS requirements for bridged dataspaces; d) Standards compliance. The design of DDS-IS is fully compliant with the OMG DDS specification, hence it is implementation agnostic. e) Performance. Conducted experiments demonstrate that the impact of the proposed service on communications performance is well within the acceptable limits for most real-world uses of DDS. The remainder of the chapter is organized as follows. In Section 3.2 we present a motivating use case scenario for the proposed service. In Section 3.3 we identify the service requirements. In Section 3.4 we describe the proposed DDS-IS architecture. In Section 3.5 we study the interactions among DDS-IS architectural elements. In Section 3.6 we include relevant implementation details. In Section 3.7 we address the conducted experiments and analyze the obtained results. In Section 3.8 we conduct a study of the impact of the service in DDS QoS policies. Finally, in Section 3.9 we summarize the main conclusions of our work and discuss some open issues that are treated in next chapters. 3.2 Motivating use case example DDS is a middleware for solving information sharing and data-exchange on realtime distributed systems. In this sense, DDS-based solutions are often deployed in industrial and mission critical applications. Two problems associated to these deployments are system heterogeneity and scalability. As systems grow in complexity, incompatibility problems arise among different software versions or among applications of different providers. In the same way, the natural evolution of enterprises generates scalability problems, and it is necessary to support bigger scenarios, perhaps even deployed among distant locations. From these problems, it emerges the need of defining DDS bridging service that improves DDS scalability and eases the interoperation of heterogeneous DDS-based applications. To identify the requirements of this service, we now detail a motivating scenario that requires the functionality of a DDS bridging service. Figure 3.1 includes an example deployment scenario for the DDS-IS. Specifically, this example shows how DDS-IS can be used in an olive oil factory. Although

87 56 Chapter 3. DDS data-space Interconnection Service (DDS-IS) Figure 3.1: Example deployment scenario. other scenarios are possible, this example illustrates the functionalities of the proposed service. The olive oil factory in our example generates data-content in two different environments, namely centrifugation system and storage system. In addition, there is a third environment (accounting and general supervision system) that stores the most relevant data-updates. Decanter centrifugation is the part of olive oil production that separates olive oil from vegetation water and solid materials. In order to produce high quality oil, operators must control the centrifugation speed, centrifuge temperature, and oil density. The centrifugation system of our example has three centrifuges. Each one publishes viscosimetric information and temperature information (with a ratio of one update per second). Each centrifuge is also subscribed to two topics: one for controlling centrifugation speed, and other for controlling the input and output valves. A local coordination node controls the decanter centrifugation system, by subscribing to temperature and viscosimetric topics, and controlling centrifuge speed and valves. Storage is the part of olive oil production where the oil is stored until its bottling and distribution. In order to conserve the quality of the oil, the system must monitor

88 3.3. Requirements 57 the pressure, temperature, humidity and light exposure of the storing tanks. In our example, we have assumed three oil tanks. Each one of these tanks publishes pressure, temperature, humidity, and light exposure information (with a rate of one update per second), and subscribes to a valve control topic (i.e., three publishers per topic for a total of 14 publishers, and three subscribers). A local coordination node controls the valve status, and monitors published metrics (using four different subscribers and one publisher). Local coordination node can react to received values if necessary (for example, by activating a refrigeration system). Although the generated data-content is mainly useful within its associated system (centrifugation or storage), the administrators of this factory want to store the most relevant events of each system in a central data-base, generating alerts if certain trigger values are reached. However, administrators do not want to modify either the current sensor deployment or the local coordination nodes logic. Moreover, it is desirable to keep the data-content of each system isolated (in order to avoid overflowing local networks with unnecessary information). Figure 3.1 presents a solution to the described scenario requirements. In particular, this solution is based on deploying two DDS-IS nodes: one for bridging and filtering data-content from the storage system (i.e., from LAN A to LAN C), and the other for bridging and filtering data-content from the centrifugation system (i.e., from LAN B to LAN C). Using this deployment, DDS-IS nodes bridge only relevant data-samples to the accounting and general supervision system, while they avoid overflowing storage and centrifugation systems with unnecessary updates. 3.3 Requirements From the previous example, we identify the following requirements that are not currently covered by DDS, and which the proposed service must fulfill: R1: Domain bridging. DDS-IS must connect different DDS data-spaces. These data-spaces are groups of publishers and subscribers that exchange data using compatible DDS middleware implementations and transport configuration. R2: Topic bridging. DDS-IS must connect different DDS topics. A topic has associated a data-type. Consequently, in order to bridge different topics our service must be able to adapt data-samples between different data-schemas. R3: Type-based transformations. In order to adapt data-samples between different data-schemas, DDS-IS must be able to perform type-based data transformations. As an additional requirement, the service must allow users for defining their own transformations.

89 58 Chapter 3. DDS data-space Interconnection Service (DDS-IS) R4: Content-based filtering. DDS-IS must be able to use samples content for determining if a sample should be bridged, or silently dropped such that it does not flow through the bridge. R5: QoS policies consistence. DDS-IS must be compatible with existent DDS QoS profiles. DDS-IS must allow administrators for establishing what QoS requirements domains must met in order to be bridged. The service must ensure that the QoS policies of the interconnected entities are fully compatible with the ones configured for the DDS-IS. In addition, DDS-IS must allow administrators for integrating dataspaces with different (perhaps even incompatible) sets of QoS. R6: Transparency. DDS-IS must not require modifications to existing DDS applications. This feature will enable legacy and new DDS applications to communicate through the proposed service. In this sense, our service will be an interoperability solution for connecting dissimilar data-spaces. R7: Standards compliance. DDS-IS must be compliant with DDS standards family. This will ensure the interoperability of the service with any DDS-compliant middleware implementation. R8: Performance and scalability. DDS-IS must have a low impact on DDS applications performance, mainly in terms of latency overhead. In addition, the system should improve DDS deployments scalability in terms of overall network traffic load. R9: Information confidentiality. DDS-IS must provide mechanisms for filtering out sensitive information from bridged data-samples. Specifically, DDS-IS must be able to use data-samples content for determining what data-samples should be discarded (this functionality is covered by R4). DDS-IS must also be able to selectively filter specific data-sample fields (instead of the whole sample) using the functionality covered by R3. In the following section we describe the architecture of our system, which satisfies the identified requirements. 3.4 Internal architecture DDS-IS architecture adopts a layered design based on the following motivations. First, it increases system modularity, by allowing the specific implementation of each layer to be changed without impacting the rest of the layers. Second, it allows existing applications to remain independent of new proposed DDS-IS entities while benefiting from the service.

90 3.4. Internal architecture 59 App/User Node 1 App/User Node 2 App1 App2 DDS-IS Node App3 App4 Application Layer Routing Engine Routing Layer Data Distribution Service Middleware Data Distribution Service Middleware Data Distribution Service Middleware Pub/Sub Layer Network Layer Figure 3.2: Layered DDS-IS architecture. And third, it simplifies the overall understanding of the system, because it classifies entities of different functionality in different layers. Figure 3.2 depicts the architecture of our solution. It can be seen as composed of a set of components located on different nodes. The components of each node are arranged in four layers, namely, application, routing, pub/sub, and network layer. In the following subsections we explain the functionality of each layer and its contained components Application layer The application layer is the topmost layer of our architecture, and it is populated by DDS applications. These applications exchange information through DDS dataspaces by means of the underlying DDS middleware. In our design, entities on this layer do not communicate directly with the routing layer. Instead, applications exchange data through the pub/sub layer using the existing standard DDS API (i.e., in the same way as if the DDS-IS was not present). This feature allows applications to remain independent of the DDS-IS, and therefore they do not require any changes. DDS-IS operation is totally transparent to the application layer. From the perspective of application layer entities, the only noticeable change is the increase in the data-content that they can access. This is because the DDS-IS enables the interoperation of applications associated with different data-spaces (i.e., different DDS

91 60 Chapter 3. DDS data-space Interconnection Service (DDS-IS) domains), virtually combining the content of the involved data-spaces. The DDS-IS also enables applications with different data models to exchange data-content. This is provided by the ability of the DDS-IS to transform data on the fly. Again, these transformations are independent of the application layer, and therefore existing applications require no modifications for benefiting from the interconnection service. As we have stated before, our solution does not modify the interactions between applications and DDS data-spaces. This is also valid for DDS compatibility checks. In this sense, if a DW (e.g. an application DW) and a DR (e.g. a DDS-IS DR) determine that they are not fully compatible (by performing the necessary DDS discovery-checks), they will not start exchanging data-content. Consequently, in order to bridge two applications with different data-types, or associated to different data-spaces, the DWs and DRs of the DDS-IS have to be properly configured. In the same way, the QoS profiles associated to bridged entities must be compatible with the QoS profiles configured for the DDS-IS. As we will discuss later in Section 3.6.5, DDS-IS administrators can configure the service for enforcing the same QoS configuration for the two interconnected data-spaces. Alternatively, system administrators can configure DDS-IS for integrating two domains with different (perhaps even incompatible) sets of QoS policies Routing layer The routing layer is the core of our design. This new layer extends the functionality of the DDS specification: it allows for bridging DDS information between different DDS domains (i.e., between DDS data-spaces). It enables information flow across different DDS topics, optionally transforming and filtering the exchanged data. As a result, from the application layer perspective, two different (perhaps even remote) data-spaces appear as a single (and extended) DDS data-space. This functionality is provided by the Routing Engine (see Figure 3.3), which contains the following components: Discovery Monitor: This component processes the discovery events generated by the underlying DDS middleware (i.e., by B-DRs). Whenever a change occurs in the DDS discovery information, the DDS middleware notifies the Discovery Monitor. The Discovery Monitor then retrieves the received discovery information from B-DRs and delivers it to the proper Domain Route. Data Engine: This component controls one or more DRs and DWs in the pub/- sub layer. The Data Engine has two missions. First, it takes received data samples from the pub/sub layer, and pushes those samples to the proper Topic Route. Sec-

92 3.4. Internal architecture 61 Routing Engine Domain Route Transformation Extensions Auto Topic Route Transformation Engine Topic Topic Topic Route Route Route Configuration and QoS Manager Discovery Monitor Data Engine Data Distribution Service Middleware Figure 3.3: DDS-IS Routing Engine. ond, it delivers resulting samples (from Topic Routes) to DWs so that they can be published. There is one Data Engine associated with each Domain Route. Domain Route: A mapping between two different DDS domains (i.e., dataspaces). A Domain Route contains multiple Auto Topic Routes and Topic Routes. The Domain Route interacts with the Discovery Monitor in order to obtain discovery information from a set of B-DRs in the pub/sub layer. After receiving discovery updates from the Discovery Monitor, the Domain Route triggers the creation of DDS and DDS-IS entities according to the service configuration. A DDS-IS instance can contain multiple Domain Route instances, each bridging a pair of DDS domains. Topic Route: A mapping between an input topic (i.e., the topic from where the DDS-IS receives data) and an output topic (i.e., the topic where the DDS-IS pub-

93 62 Chapter 3. DDS data-space Interconnection Service (DDS-IS) lishes data). Therefore, a Topic Route is defined by an input topic name and type, and an output topic name and type. Additionally, a Topic Route can apply transformations to data-samples using the Transformation Engine (explained below). The behavior of a Topic Route is as follows: first, it receives data from its associated Data Engine; then, it applies transformations (if any); and finally, it delivers processed data to the Data Engine for publication. A Domain Route instance can contain multiple Topic Route instances, each one bridging a pair of DDS topics. Transformation Engine: This element receives input data samples from a Topic Route, applies a set of transformations to those samples, and returns the resulting data samples. If a Topic Route requests for multiple transformations, the Transformation Engine processes them in sequence and following a predefined order (this is important because the output of a transformation and the input of the subsequent transformation must be type-compatible). Supported transformations are defined through multiple Transformation Extensions. Transformation Extension: This element receives input data samples, applies a transformation to those samples, and returns the result of the transformation. To be able to process data of any type, Transformation Extensions use DDS Dynamic Topic Types. In order to increase system flexibility, A DDS-IS instance can contain multiple Transformation Extension instances, each one describing a particular data transformation. Transformation Engine can load external Transformation Extensions implemented by DDS-IS users. Auto Topic Route: A generic Topic Route that bridges a range of topics. The topic name and type for bridged topics need not be known a priori, and therefore Auto Topic Route provides auto-configurable topic bridging. Auto Topic Routes are configured with a subset of topic names and/or topic types for which the automatic setup is allowed. A Domain Route instance can contain multiple Auto Topic Route instances, each one covering a different set of topics. Configuration and QoS Manager: Following the philosophy of the DDS standard, our design relies on a set of QoS policies for configuring the behavior of routing layer elements. Indeed, some of these QoS policies also control the configuration of entities belonging to the DDS core layer (for example, Topic Route QoS profiles also control the configuration of the associated DW and DR). The Configuration and QoS Manager is the component that controls the proper configuration of the DDS-IS Pub/Sub and network layers The pub/sub layer (see Figure 3.2) contains the entities associated to the DDS middleware (described in Section 2.2.1), and interacts with system s upper layers using

94 3.5. System operation 63 the standard DDS API. The pub/sub layer allows the DDS-IS to access to DDS data-spaces. In order to provide this functionality, the pub/sub layer perform the following tasks: (1) it notifies the routing layer about changes in discovery information, (2) it receives new data samples and delivers these samples to the routing layer, and (3) it publishes data obtained from the routing layer to DDS data-spaces. In addition to the entities already introduced in Section 2.2.1, this layer includes two DDS WaitSets. A DDS WaitSet is an standard DDS entity that can be used to wait for specific DDS Events. DDS WaitSets are awaken whenever certain predefined trigger conditions are met. Specifically, our system uses the following two DDS WaitSets: Discovery WaitSet: This element receives updates from B-DRs (see Section 2.2.1). It notifies the Discovery Monitor in the routing layer when a change occurs to discovery. This allows the DDS-IS to react to the appearance and disappearance of entities to/from the connected data-spaces. Data WaitSet: This element notifies its associated Data Engine in the routing layer about the reception of new data samples in one of its DRs. This allows the DDS-IS to process DDS samples from the DDS core layer as soon as they are received. The network layer is the lowest layer of the proposed architecture (see Figure 3.2). It represents the platform-dependent components that are necessary for exchanging data-samples (such as the operating system, computer hardware, or communication network). One of the advantages of using a network middleware (such as DDS) is that it decouples the application logic from the particularities of the network layer. Consequently, we do not need to define any requirements for this layer, except that the chosen DDS implementation must be DDS-RTPS-compliant. 3.5 System operation In order to illustrate the interactions among the previously described entities, in this section we provide sequence diagrams for the two main system use cases.

96 3.5. System operation 65 gers the initialization of a Topic Route and all its associated entities. A B-DR initiates the process when it receives a change in the discovery information of a DDS entity located in the same data-space (but associated to a different node). This event triggers a Discovery WaitSet (studied in Section 3.4.3), which then sends a notification to the Discovery Monitor. Once the Discovery Monitor receives a discovery notification, it retrieves the nature of the originating B-DR (participant, publication or subscription), and the discovery information received by the B-DR. At this point, the Discovery Monitor delivers all the received discovery samples to the Domain Route associated to the originating B-DR. Received samples can contain multiple changes in the discovery, which are processed separately. These changes can refer either to the creation or removal of remote DDS entities. Using the received information, the Domain Route updates an internal register that stores information of previously discovered DDS entities and topics. This register enables the DDS-IS for determining if it is necessary to create new Topic Routes upon the reception of discovery information. Then the Domain Route checks if any of its active Auto Topic Routes matches with the received discovery information. If necessary, the Domain Route creates new Topic Routes according to Auto Topic Routes configuration. Finally, the Domain Route checks the existing Topic Routes, and then determines if the discovered DDS entity is associated to any of them. If the Domain Route finds a matching Topic Route, the Domain Route will check if the Topic Route is already active. If the Topic Route is not active, the Domain Route starts the Topic Route (i.e., the Topic Route requests the Data Engine to create the necessary DR and DW for performing topic bridging) Data bridging Data Bridging is the process of communicating data-samples between two different DDS data-spaces. If the input and output data-spaces share the same topic, the DDS-IS can bridge data-samples without performing any modifications to datasamples structure. However, the DDS-IS can also bridge data-samples between two data-spaces that do not share the same data-model. In this case, the DDS-IS must transform input data-samples for adapting them to the format required at the output data-space. Figure 3.5 depicts the sequence diagram of the data bridging process. A DR (located in the pub/sub layer) initiates the process when it receives a new data-

97 66 Chapter 3. DDS data-space Interconnection Service (DDS-IS) Pub/Sub Routing DDS Data Engine Topic Route Transformation Engine Transformation Extension Notify Data Reception Get Data Deliver Data to proper Topic Route Apply Transformations Apply Transformation Publish Data Publish Data Figure 3.5: Data bridging sequence diagram. sample from a DW located in the same data-space (but in a different node). This event triggers a Data WaitSet, also located in the pub/sub layer, which then sends a notification to the Data Engine associated to the DR. After receiving the notification from DDS, the Data Engine retrieves all of the received samples from the originating DR, and delivers those samples to the Topic Route associated to that DR. At this point, the Topic Route applies configured transformations (if any) to the input data-samples (using the Transformation Engine). Data Bridging then finishes with the publication of resulting data-samples to the output data-space by means of the proper Data Engine s DW (located in the pub/sub layer).

98 3.6. Implementation Implementation After defining the DDS-IS architecture and its design, we have implemented a proofof-concept prototype. This prototype was used to validate the proposed design, to perform field-testing, and to evaluate the performance impact of our service (for further information regarding conducted experiments and obtained results, please refer to Section 3.7). In this section, we explain the most relevant implementation decisions that we have taken during the prototype development Dynamic types As we studied in Section 2.2.4, the DDS-XTypes API allows DDS applications to publish and subscribe topics with types unknown at the time the application was compiled. Since DDS-IS needs to interact with DDS applications not known a priori (including DDS applications to be developed in the future), the DDS-IS makes an extensive use of the DDS-XTypes API in order to discover and manipulate previously unknown data types, including performing any necessary data transformations DDS middleware implementation The DDS-IS prototype was implemented using Real Time Innovations Inc. (RTI) s DDS version 4.4d [88]. We have chosen this implementation because it supports the DDS Dynamic Topic Types extension. This extension is a new API for the DDS standard family that was recently standardized by the OMG [70]. Nevertheless, our design is implementation-independent, as it only uses standardized DDS features plus the DDS Dynamic Topic Types extension. Consequently, the proposed design can be implemented using any other DDS-compliant implementation. Indeed, the implemented prototype is also compliant with the DDS-RTPS [69]. Regarding the used programming language and bindings, we have implemented our prototype in C, and we have compiled the source code using gcc version DDS signaling propagation As we have mentioned, a Topic Route connects an input data-space s topic (i.e., a set of application DWs) with an output data-space s topic (i.e., a set of application

99 68 Chapter 3. DDS data-space Interconnection Service (DDS-IS) DRs). To enable the interoperation of these DWs and DRs, the DDS-IS must create both a DR at the input domain and a DW at the output domain. In this scenario, the question naturally arises: when should the service create the input/output DR/DW? The answer to this question depends on the particular application requirements. If system resources (memory and CPU) are scarce, DDS-IS should not create the paired DW-DR until a match between the input topic and the output topic is found (i.e., both the matched DR and DW have been discovered). However, if we need all of the DDS signaling information (i.e., DDS discovery information) to propagate through the DDS-IS, the service should create both the DR and DW as soon as a matched publication or subscription is discovered. Alternatively, it may be preferable for the system to set up Topic Route s entities as soon as the user configures a Topic Route (even if the DDS-IS does not find any matching DWs or DRs). In order to support scenarios with different requirements and constraints, we decided that the prototype should allow the user to configure the triggering conditions for the creation of Topic Route s entities. In Section 3.6.6, we provide an example of how to configure the creation for the input and output of a Topic Route Propagating DataWriter information By default, DDS middleware generates an unique IDentifier (ID) (using hosting machine information and implementation specific parameters) for each DW. In the early stages of DDS-IS prototype implementation, this was also the behavior of the service. However, after some testing we concluded that this behavior prevents application DRs from distinguishing between original data-samples and replicas of those data-samples. The ability of distinguishing between original data-samples and their replicas is especially critical for deployments with redundant DDS-IS nodes. In these scenarios, this functionality may significantly impact system resources consumption, given that it avoids the delivery of duplicated samples to final applications. Consequently, we decide to enable the DDS-IS to maintain the original DW information, which is contained in input data-samples. The implemented prototype allows configuring if the original DW information is maintained, or if it is replaced by the DDS-IS DW information. In this way, the DDS-IS administrator is able to configure the system behavior according to the specific scenario requirements.

100 3.6. Implementation Quality of Service As we described in Section 3.4.2, DDS-IS relies on DDS QoS profiles for configuring the behavior of the service. Consequently, DDS-IS also relies on DDS built-in compatibility-checking mechanisms for ensuring that QoS profiles of communicating entities are fully compatible. DDS-IS QoS checking is performed on a hop-by-hop basis (i.e., between DDS-IS entities and application entities located within the same domain). This feature allows system administrators for decoupling QoS for different domains, enabling the integration of heterogeneous (and perhaps even incompatible) scenarios. Of course, administrators can configure the same QoS policies for the two interconnected domains, which guarantees that QoS remain invariant across interconnected data-spaces. Regarding QoS enforcement, DDS QoS policies were initially intended for working within single domains. Consequently, the behavior of DDS QoS policies in DDS-IS-based deployments is currently unknown. In this regard, in Section 3.8 we include an analysis of the impact of DDS-IS on each DDS QoS policy XML configuration format The DDS-IS uses extensible Markup Language (XML) files to load its configuration. XML is a set of rules for encoding documents. These rules are defined in the XML 1.0 Specification produced by the W3C [106]. Figure 3.6 provides an example of a basic XML configuration file; specifically, it includes the following XML tags: dds: The root of the XML document; it can also include the definition of QoS policies for the system. dds is: This tag includes the configuration of a DDS-IS instance. every other DDS-IS should be included as a child of the dds is tag. Therefore, domain route: This contains the configuration for a Domain Route and must contain a participant 1, a participant 2, and a session tag. participant 1 and participant 2: These tags define the input and output DDS domain for the service using the domain id tag. session: This contains the configuration for a set of Topic Routes and Auto Topic Routes that share the same Data Engine instance. Session tag can contain multiple topic route and auto topic route tags.

102 3.7. Validation and performance evaluation 71 auto topic route and topic route: The system uses these tags to set up the Auto Topic Routes and Topic Routes. publish with original info: This determines if the DDS-IS maintains the original DW information for publishing data to the output. publish with original timestamp: This determines if the DDS-IS maintains the original timestamp information for bridged data-samples. input: This defines the configuration of the route input. It allows configuring filters for topic routing, the QoS policies for the input DR, and the creation mode, which is defined below. output: This defines the configuration of the route output. It allows configuring filters for topic discovery at the output, the QoS policies for the output DW, and the creation mode, which is defined below. creation mode: This configures the way discovery information is propagated from the input to the output (or vice versa). In this way, when a DR is discovered at the output, the system can set up the corresponding Topic Route immediately (as in the example) or wait until a DW is discovered at the input. 3.7 Validation and performance evaluation To study the operation and performance of the DDS-IS prototype, we conducted a set of experiments in a controlled LAN environment. Specifically, we measured the impact of the DDS-IS in terms of round-trip latency, throughput, and network traffic load Experimental set-up The test-bed was composed of six Core i5 at 2.40GHz-2.66GHz machines (named lab01, lab02, lab03, lab04, lab05, and lab06) running Linux Kernel x86 64 (Ubuntu 10.04) and the RTI DDS 4.4d middleware. These machines communicate through a 24-port Gigabit switch with Virtual Local Area Network (VLAN) support. In our experiments, these nodes receive one of three possible roles: Source-node: A node that generates DDS samples, receives responses for those samples, and calculates experimental statistics. Echo-node: A node that receives DDS samples from source-nodes and generates responses.

103 72 Chapter 3. DDS data-space Interconnection Service (DDS-IS) Routing-node: A node that bridges two different networks. We developed a benchmarking tool for conducting our experiments. This benchmarking tool consists of two components: PerformanceTest tester: This application runs in the source-nodes. It publishes samples into a topic called Ping and receives them from a subscribed topic, called Pong. Ping samples contain a sequence number, the publication timestamp, and a variable size payload as well. The structure complexity of Ping payload is variable. Pong samples only contain a sequence number and the original Ping publication timestamp. This timestamp is compared with the sample reception time for obtaining samples round trip time. Each Ping and Pong sample is sent in a separate message. PerformanceTest remote: This application runs in echo-nodes. It takes samples from the Ping topic and republishes their sequence number and timestamp field content into the Pong topic. In the conducted tests, published samples performed a round trip between source-nodes and echo-nodes. This approach allows for the precise measurement of the sending and reception times without performing any clock synchronization. Consequently, at least two different routes have to be configured in the DDS-IS: the outgoing route (for bridging Ping samples) and the incoming one (for bridging Pong samples). For all the reported evaluations, our benchmarking tool uses RTPS over UDP protocol, and sends each sample separately (i.e., each UDP datagram contains only one RTPS sample). Regarding sample sizes, Ping samples contain a payload whose size ranges from 256 to 8192 bytes and a protocol overhead of 118 bytes. Pong samples have the same protocol overhead as Ping samples, but their payload has a size of 12 bytes because they only contain a sequence number and a timestamp. We considered the following metrics for the DDS-IS performance evaluation: Throughput: The average number of samples per time interval (seconds) received from the data-space by subscribers. Round-trip Latency: The average end-to-end elapsed time for delivering samples from the original source-node to the echo-node plus the elapsed time for delivering samples back to the original source-node. Our tool measures the latency at application level. In this sense, publishers measure the time before delivering data-samples to DDS middleware, whereas subscribers measure the time after DDS delivers data-samples to the benchmarking tool. Generated traffic load: The total generated traffic (in bytes) for a specific net-

104 3.7. Validation and performance evaluation 73 work segment in a particular scenario. This includes the payload plus all the overheads generated by the whole protocol stack. In order to evaluate the proposed service performance, we conduct all the experiments in two different scenarios. Namely, No-DDS-IS Scenario: Publishers and subscribers associated to source-nodes and echo-nodes were located in the same data-space, but in different networks. DDS entities communicate through one or more Layer-3 Linux Routers. DDS-IS Scenario: Publishers and subscribers associated to source-nodes were located in a different data-space (also in a different network) than the ones associated to echo-nodes. DDS-IS interconnects the involved data-spaces. In both scenarios, samples were published using the DDS Reliability QoS policy set to RELIABLE, which guarantees that the data sent by DWs is received by DRs. Additionally, we have configured the QoS policy KEEP_HISTORY to ALL_HISTORY, which guarantees no sample is dropped from DDS buffers. This configuration will force the DDS middleware to ensure the delivery of all the samples. However, it can cause that few samples have an extremely high latency, potentially biasing the average results. In this respect, the reported results only include the 95th percentile latency. As part of the RELIABLE Reliability QoS configuration, DW send heartbeat messages to their DR. Each heartbeat message contains the sequence number of the most recent sample sent by the DW. We have set DDS DR heartbeats low-period (i.e., the one used when DRs cache has less than 5 samples) to 100 milliseconds, and we have set DDS DR heartbeats high-period (i.e., the one used when DRs cache has more than 15 samples) to 10 milliseconds. We have also configured the DDS middleware for discovering DDS entities only in the interfaces of interest for each particular experiment. In this respect, all the experiments use a secondary network for launching the benchmarking tools without interfering on the tests. As we explained in Section 3.6, the DDS-IS prototype has been implemented using RTI s C API. On the other hand, the benchmarking tool has been implemented using C++. Regarding Linux kernel configuration parameters, we have used the default values associated to Ubuntu for all the machines.

105 74 Chapter 3. DDS data-space Interconnection Service (DDS-IS) ping ping pong pong LAN A LAN B Data Space 1 Data Space 2 Figure 3.7: DDS-IS scenario for 1 to 1 performance experiments. In the N to N case, lab01 and lab03 run multiple publishers and subscribers DDS-IS Impact in N to N communications In this section, we evaluate the impact of the DDS-IS in terms of round-trip latency and throughput for different payload (sample sizes), data types, and DDS-IS configurations when N publishers and N subscribers communicate following a 1 to 1 relationship. To make the results comparable, both no-dds-is and DDS-IS scenarios used the same network topology: two LAN networks (A and B) connected through a computer (lab02) with two Ethernet interfaces. In the no-dds-is scenario, lab02 is a single layer-3 Linux Kernel Router (we use just one data-space), whereas in the DDS-IS scenario, lab02 interconnects two different data-spaces and provides the proposed DDS-IS (see Figure 3.7). Each test consisted of publishing 500,000 samples per topic and using a specific payload size. These samples were published at source-node (lab01) into the Ping topic; after being received by Ping subscriber at echo-node (lab03), they were published back using the Pong topic. To avoid transitory behavior (warm-up) effects, before starting the tests we force some samples to be exchanged after the DDS entities discovery.

106 3.7. Validation and performance evaluation DDS-IS 1 to 1 No DDS-IS 1 to DDS-IS 1 to 1 No DDS-IS 1 to Latency (microseconds) Throughput (samples/s) ST 512B CT 512B CT 1024B ST 1024B CT 1024B ST 1024B CT 512B ST 512B Figure 3.8: DDS-IS impact for different data types in 1 to 1 communications Performance impact in simple 1 to 1 communications This first set of experiments aims at the evaluation of the performance impact for different data types and DDS-IS configurations when only one topic for Ping and one for Pong is active. Although the Pong topic was always the same, this experiment uses the following Ping topic types: SimpleType (ST) 512, 1024 bytes: These contain a sequence of random bytes, a sequence number, and a timestamp. ComplexType (CT) 512 bytes: This structure contains two levels of nesting. It includes a SimpleType (256 bytes) data, four 64byte-string fields (two of them in a nested struct), a long field, and eight Boolean fields. ComplexType (CT) 1024 bytes: This structure contains three levels of nesting. It includes two ComplexType-512 byte structs. Figure 3.8 shows the performance results of both no-dds-is and DDS-IS scenarios. In this experiment neither transformations nor filtering are enabled. The results show that although the DDS-IS introduces approximately a roundtrip latency overhead of 250 microseconds, it has no a significant impact on through-

107 76 Chapter 3. DDS data-space Interconnection Service (DDS-IS) DDS-IS 1 to DDS-IS 1 to Latency (microseconds) Throughput (samples/s) CT 1024B filt. CT 1024B transf. CT 1024B CT 512B filt. CT 512B transf. CT 512B CT 1024B filt. CT 1024B transf. CT 1024B CT 512B filt. CT 512B transf. CT 512B Figure 3.9: DDS-IS performance for filtering and transforming data. put. We can also conclude that in this case data complexity does not have a significant performance impact on DDS-IS scenario in comparison to no-dds-is scenario. Consequently, in the remainder experiments we focus our study on testing different sample sizes Performance impact of data transformation and filtering features The DDS-IS allows loading external data transformations and applying these transformations to the output data. In this sense, the performance degradation is highly dependent on the complexity of the transformation. Therefore, rather than reporting an exhaustive analysis, this section aims to evaluate the cost of using simple data transformations, which will show the overhead introduced by inclusion of an additional function call to the datapath. Figure 3.9 shows the incurred latency and throughput for a simple data trans-

108 3.7. Validation and performance evaluation 77 formation definition (labeled as CT size transf.). In particular, we have set up the DDS-IS for changing a long field and a Boolean field to a specific configured value. In this experiment, we observe that the complexity of the transformed structure degrades the throughput (see the obtained results for labels CT 512B transf. and CT 1024B transf.). However, simple transformations have no significant negative impact on latency. Indeed, for CT512 samples, using transformations reduces the average latency. This behavior is because the additional processing time introduced by transformations reduces the number of context switches in the routing-node Central Processing Unit (CPU), what reduces the average latency overhead. To test impact of data filtering we configured a filter that drops samples according to the content of the samples. Published samples are labeled with a number randomly obtained from a uniform distribution for the interval 0 to 999. The DDS-IS filters the samples according to the following definitions. For CT512 case (labeled in Figure 3.9 as CT 512B filt.) the filter was: (long_cond_00 < 500) or (nested_ping.sequence_number = ) Whereas for CT1024 case (labeled in Figure 3.9 as CT 1024B filt.) it was: (nested_complexping_02.long_cond_00 < 500) or (nested_complexping_01.nested_ping.sequence_number = ) The first condition on each filter drops approximately the 50% of the samples. The second condition ensures the warm-up signaling samples traverse the DDS-IS filter. According to the obtained results (see Figure 3.9), the filter introduces an average overhead of 110 microseconds when it uses a filtering condition on the first level of the type CT512, and 140 microseconds for evaluating the condition in a second level of nesting of the CT1024. In addition, DDS-IS approximately halves the throughput compared to a scheme without filtering, as expected from the fact that this filter drops 50% of the samples. This experiment validates the filtering and data transformation features of the DDS-IS. We can conclude that for the evaluated conditions DDS-IS accomplishes these tasks without a significant degradation of system performance Performance impact in N to N communications with multiple topics In this section, we study how DDS-IS affects performance as the number of concurrent distinct topics (and Topic Routes) increases.

109 78 Chapter 3. DDS data-space Interconnection Service (DDS-IS) In this case, we use the same topology of Figure 3.7 but now we increase the number of bridged topics (up to 16). We always use an even number because our benchmarking tool utilizes two types of topics: Ping for outgoing samples and Pong for incoming ones. This experiment is a worst-case scenario for the DDS-IS operation, as the capabilities of data multiplexing of DDS-IS are not used. Indeed, the DDS-IS has to maintain a pair of DR/DW per route. More particularly, the last case studied in this section creates 16 DWs and 16 DRs in the same DDS-IS. Although this is not an appropriate scenario of DDS-IS operation (multiple 1-1 routes), these experiments provide some insight on the DDS-IS limits. Figure 3.10 depicts the average round-trip latency as a function of the payload size. The results show that the DDS-IS performs reasonably well for samples with a size up to 6144 bytes when 2 to 16 concurrent routes (topics) are active. Specifically, the DDS-IS introduces an overhead from 250 to 400 microseconds for the round-trip latency. For the 8192-sized payload, the scenarios with 8 and 16 concurrent topics reveal a performance degradation. In these cases the round-trip latency overhead introduced by the DDS-IS increases to 450 microseconds and 1.3 ms respectively. Additionally, the standard deviation for round-trip latency reaches 2 ms, which indicates the DDS-IS performance degrades for this load level. Regarding throughput, the results in Figure 3.11 show that DDS-IS has an insignificant impact on the throughput for the scenarios with 2-4 concurrent topics. For the 8 and 16 topics cases, the DDS-IS throughput achieves at most 30,000 samples per second (for the 64-byte payload), whereas the scenario without DDS-IS reaches 50,000 samples per second in the 8-topic case, and almost 75,000 samples per second in the 16-topic case (both for the 64-byte payload). It is important to remark that in the 16-topic case the DDS-IS is publishing 30,000 samples in both directions (Ping samples from source-nodes to echo-nodes and Pong samples from echo-nodes to source-nodes). This is especially relevant in the 64-byte scenario, in which the size of Pong samples is similar to the size of Ping samples. In this scenario, while source-nodes and echo-nodes just manage one DW (for publishing Ping samples or for publishing Pong samples) and one DR (for receiving Pong samples or for receiving Ping samples), the routing-node has to manage a pair of DW/DR per topic. We have also measured the impact on CPU consumption for the routing-node (i.e., for lab02). Figure 3.12 shows that the maximum CPU consumption is obtained for the experiments using small packets, which are the ones with a highest rate of

111 80 Chapter 3. DDS data-space Interconnection Service (DDS-IS) CPU % Data Size (Bytes) 2 Topics 4 Topics 8 Topics 16 Topics Figure 3.12: CPU impact for different payloads and number of routes (topics) active. samples per second. The obtained results also show that the degradation in performance for bigger sizes is not due to CPU saturation. Instead, it is mainly due to synchronization problems between the middleware and the used Ethernet interfaces, along with the increased cost of packet disassembling, assembling, and loss-recovery DDS-IS impact in M to N communications In the previous section, we studied the limits of the DDS-IS in a scenario where no data multiplexing is present. The following experiments analyze the performance of the DDS-IS when it delivers several copies of certain published data to multiple subscribers.

112 3.7. Validation and performance evaluation 81 ping pong pong ping ping LAN A LAN B ping ping Data Space 1 Data Space 2 Figure 3.13: System topology used in 1 to N performance experiments DDS-IS performance impact Figure 3.13 shows the topology used in this set of experiments. Specifically, we run a Ping topic publisher and a Pong topic subscriber on node lab01, a DDS-IS in node lab02 (for the no-dds-is scenario, IP forwarding was enabled at this node), and 0-4 subscribers in each of the nodes lab03, lab04, lab05, lab06. To avoid overloading the DDS-IS with Pong samples, only one Pong publisher at node lab03 publishes Pong samples back to lab01 (for measuring average round-trip latency and throughput on lab01 node). Figure 3.14 depicts average round-trip latency results obtained for this set of experiments. The obtained maximum overhead is 330 microseconds (for the bytes payload, 1 to 1 experiment). For the 4096-bytes, 1 to 4 case, the DDS-IS scenario obtains almost zero round-trip latency overhead (852 and 846 microseconds for the cases with and without DDS-IS respectively). Indeed, for a number of subscribers greater than 4, and a payload greater than 4096 bytes, we have obtained lower round-trip latencies values for the DDS-IS scenario than for the no-dds-is one. This is because the latency overhead introduced by the DDS-IS logic is offset by the reduction of the load on lab01 (the original publisher), the reduction of traffic in LAN A (this will be studied in next section), and

113 82 Chapter 3. DDS data-space Interconnection Service (DDS-IS) Latency (microseconds) No DDS-IS 1 to 1 DDS-IS 1 to 1 No DDS-IS 1 to 4 DDS-IS 1 to 4 No DDS-IS 1 to 8 DDS-IS 1 to 8 No DDS-IS 1 to 16 DDS-IS 1 to Payload size (bytes) Figure 3.14: Average latency for different payloads when demultiplexing data from 1 to N. Throughput (samples/s) No DDS-IS 1 to 1 DDS-IS 1 to 1 No DDS-IS 1 to 4 DDS-IS 1 to 4 No DDS-IS 1 to 8 DDS-IS 1 to 8 No DDS-IS 1 to 16 DDS-IS 1 to Payload size (bytes) Figure 3.15: Throughput per subscriber for different payloads when demultiplexing data from 1 to N.

114 3.7. Validation and performance evaluation 83 ping ping ping ping ping pong pong pong ping LAN A LAN C LAN B ping ping Data Space 1 Data Space 2 Data Space 3 Figure 3.16: System topology used in M to N scalability experiments. the fact that it is more efficient to re-send lost samples from lab02 (in the same LAN) than from lab01 (located at two hops). Regarding throughput, the results in Figure 3.15 show that the DDS-IS scenario beats the no-dds-is scenario for all the experiments with a number of subscribers greater or equal than 4. This is especially relevant for the 64-byte, 1 to 16 scenario, where the DDS-IS maintains a rate of 10,000 samples per second per subscriber, while in the no-dds-is case the rate drops to 5400 samples per subscriber. The explanation for this behavior is the same we stated for latency experiment: network usage reduction on LAN A, reduction of load for original source-node, and the fact that re-sending from lab02 is cheaper than re-sending from lab DDS-IS scalability impact Earlier in this chapter we stated that the DDS-IS can help in reducing the network traffic load. The following experiment (Figure 3.16) supports our claim. The experimental set-up consists of 1-4 Ping publishers (running on lab01), 1-15 Ping subscribers (uniformly distributed among lab04, lab05, and lab06), and two DDS-IS (running on lab02 and lab03). To avoid overloading the DDS-IS with Pong samples, only one Pong publisher at node lab04 publishes Pong samples back to lab01.

Management effectiveness evaluation: for the CBD and for better parks Principles for MEE Methodologies Key question: How will the evaluation help management? Before choosing a methodology or undertaking

Resumen de Entrevista: Asociación de Agentes de Aduana del Puerto de Manzanillo 1. To your knowledge, to what extent do customs brokers run into operative inconveniences when it comes to collecting payment

What is the Common Problem that Makes most Biological Databases Hard to Work With, if not Useless to most Biologists? RUNI VILHELM MRAG Americas, Inc. 110 South Hoover Blvd., Suite 212 Tampa, Florida 33609-2458

Guía Integrada de Actividades Contexto de la estrategia de aprendizaje a desarrollar en el curso: The activity focuses on the Task Based Language Learning (TBLL). The task is used by the student in order

Parents What you need to know about college admission tests Your child will want to take a college admission test, such as the SAT or other college entrance exams, when he or she is a junior or senior.

Another academic year just started and HETS wishes you success in all your endeavors during this year. This is the time for all our resolutions to take place in producing results for our learners. We have

Survey Open-Ended s Open-Ended s Parent Survey for schools Question 1. What do you like best about our school? s. The learning enviroment and the individual care for each child needs IB program the good

PREDICTING THE INTENTION OF USING A TECHNOLOGY PRODUCT OR SERVICE THROUGH A FUZZY CONTROLLER Libertad Caicedo Acosta (District University Francisco José de Caldas, Bogotá, Colombia) licedoa@correo,udistrital.edu.co

General Certificate of Education Advanced Level Examination June 2014 Spanish Unit 4 Speaking Test Candidate s Material To be conducted by the teacher examiner between 7 March and 15 May 2014 (SPA4T) To

Dear Parent, Today was the 100th Day of School, and what better way to celebrate than with activities all about the number 100? With the help of Peg and Cat the problem-solving, math-loving duo from PBS

When to select APA and other Transfer Pricing options Challenges and Opportunities for the Maquila Industry Agenda 1. Background 2. Type of Maquiladoras 3. APA as an option for PE exemption and compliance

Universidad de Murcia Department of Information and Communication Engineering Development of distributed algorithms for data search and content distribution in structured peer-to-peer networks A thesis

International Civil Aviation Organization SAM/AIM/3-WP/05 South American Regional Office 28/02/12 Third Multilateral Meeting of the SAM Region for the Transition of AIS to AIM (SAM/AIM/3) Lima, Peru, 12

FOR TEACHERS ONLY The University of the State of New York REGENTS HIGH SCHOOL EXAMINATION S COMPREHENSIVE EXAMINATION IN SPANISH Wednesday, January 24, 2007 9:15 a.m. to 12:15 p.m., only SCORING KEY Updated

COMPUTER CLASS REGISTRATION FORM (Please Print Clearly Lea con cuidado) To register for the Computer Technology Program, please complete the following form. All fields in this form must be filled out in

Estrategias para la Reducción de Riesgos y Ciber Ataques Luis Zamora Consultor en Tecnología 1 This document is for informational purposes. It is not a commitment to deliver any material, code, or functionality,