Recent improvements to the World Wide Web and wireless technologies have led to aplethora of ideas and concepts that either improve upon existing implementations or areoriginal altogether. In this thesis, we propose a novel approach for an overall teleroboticsystem for controlling the iRobot Create® mobile robot that will enhance a user’s experienceby being intuitive to use, ergonomically friendly, accessible, portable and scalable. Thisnovel concept involves the creation of Robajax; a Web application built using an AJAXmethodology. Previous applications for telerobotic control were PC based applications thathad to be installed as a monolithic desktop application. Telerobotic control involves a seriesof mouse clicks to direct a robot’s trajectory. The Robajax Web User Interface, being a Webapplication, renders local installation unnecessary, as the Robajax Web User Interface isaccessible to anyone over an Internet connection. Mouse clicking to direct the robot to moveis not needed as the user interface tracks mouse movement in a circular widget to directmotion. Thus, users can easily control the iRobot Create through obstacles for an extendedperiod of time without witnessing hand fatigue, making the Robajax Web User Interfacemore ergonomically friendly when compared to a PC application counterpart. This overallsystem uses DHCP to acquire a dynamic IP address for the iRobot. A dynamic IP addressallows this system to be portable across different wireless networks because the iRobot canbe placed on any wireless network, including wireless free-Wi-Fi hotspots, and still connectto the Robajax Web Application Gateway. Scalability is solved by using the Robajax WebApplication Gateway Web Service. The Robajax Web Application Gateway maintains a one-to-many TCP session relationship with the Web browsers of remote users and a one-to-onesession relationship between the iRobot and the Web Application Gateway. The purpose ofrestricting the number of sessions over the WLAN to a single session is to prevent thestreaming of redundant video to multiple parties, thus allowing the wireless channel to carrya single video stream to the gateway with the highest possible frame rate or lowest possiblequantization value, depending on the available wireless data rate. Using this design approach,the Robajax Web User Interface we have developed is scalable, portable, accessible andergonomically friendly. Design details regarding the use of AJAX, multithreaded SOAP-based Web Services, and algorithms used to prevent deadlock conditions in Web-basedtelerobotic control are presented.

Figure 2.5. The iRobot will generate more traffic if the Robajax Web ApplicationGateway is not used. ....................................................................................................18

Figure 3.6. A SOAP message is sent immediately when the mouse enters the redcircle. ............................................................................................................................26

Figure 4.6. The iRobot is now the connection initiator to adapt to SDSU firewallrestrictions. ...................................................................................................................33

xiACKNOWLEDGEMENTSI would like to thank my family for their encouragement and support. Without theirencouragement I would have never would have pursued an advanced degree. I would alsolike to Dr. Christopher Paolini for giving me guidance and supervision throughout this thesis.Furthermore, I would also like to thank Dr. Bill Root and Dr. Carl Eckberg whose courseshave helped me immensely to complete this thesis.

1CHAPTER 1INTRODUCTION TO TELEROBOTICSIn this thesis, we developed a telerobotic system for controlling the iRobot CreateTM

Vehicle® consumer robot appliance [1]. This chapter introduces the acronyms andterminology that will be used throughout this thesis and define what telerobotic control is anddiscuss previous work on telerobotic control.1.1

ACRONYMS ANDTERMINOLOGY

User Interface (UI) – A place where interaction between humans and machines occurs. Thegoal of interaction between human and a machine at the user interface is effective operationand control of the machine, and a feedback from the machine which aids the operator inmaking operational decisions.Graphical User Interface (GUI) – A type of user interface item that allows people tointeract with programs in more ways than typing; with images rather than text commands. AGUI offers graphical icons and visual indicators, as opposed to text-based interfaces, typedcommand labels or text navigation to fully represent the information and actions available toa user.Widget – An element of a GUI that displays an information arrangement changeable by theuser, such as a window or a text box. The defining characteristic of a widget is to provide asingle interaction point for the direct manipulation of a given kind of data. Widgets are basicvisual building blocks which, combined with an application, hold all the data processed bythe application and the available interactions on the data.HyperText Markup Language (HTML) – Predominate markup language for web pages.HTML provide a means to create structured documents by denoting semantics for text suchas headings, paragraphs, lists, links, quotes and other items. It allows images and objects tobe embedded and can be used to create interactive forms.

2Extensible Markup Language (XML) – A set of rules for encoding documents in amachine-readable form; XML was originally designed to emphasize simplicity, generalityand usability over the Internet.Web Services Description Language (WSDL) – Is an XML-based language that provides amodel for describing Web Services. The WSDL define services as a collection of networkendpoints, or ports. The WSDL specification provides an XML format for documents for thispurpose. The abstract definitions of ports and messages are separated from their concrete useor instance, allowing the reuse of these definitions.Simple Object Access Protocol (SOAP) – A protocol specification for exchangingstructured information in the implementation of Web Services in computer networks. SOAPuses XML as its message format.JavaScript Object Notation (JSON) – A lightweight text-based open standard designed forhuman-readable data interchange.Web Service – Is typically an application programming interface (API) that is processed viaHTTP and executed on a remote system hosting the requested service. Web Services tend tofall into one of two camps: "big Web Services" or SOAP based Web Services andRepresentational State Transfer (RESTful) Web Services. A "Big Web Service" uses XMLmessages that follow the SOAP standard and have been popular with traditional enterprises.In such systems, there is often a machine-readable description of the operations offered bythe service written in WSDL.JavaScript – A scripting language typically used to enable programmatic access tocomputational objects within a host environment. JavaScript is primarily to provide enhanceduser interfaces and dynamic web websites.XMLHttpRequest (XHR) – An API available in web browser scripting languages such asJavaScript. XHR is used to send Hypertext Transfer Protocol (HTTP) or Hypertext TransferProtocol Secure (HTTPS) requests directly to a web server and load the sever response datadirectly back into the script.Asynchronous JavaScript and XML (AJAX) – A methodology to develop webapplications, this methodology consists of using cascading style sheets (CSS), dynamicdocument object model (DOM), asynchronous XHR and the use of JavaScript to manipulatethe DOM [2]. Web applications developed using AJAX methodology places less strain on the

3server when compared to conventional web applications written on complete HTML pages.This is because, user actions on conventional web applications causes the entire page to bere-loaded by the server. User actions with web applications using AJAX only re-load thechanged information.GlassFish – An open source application server developed by Sun Microsystems for the JavaEnterprise Edition Platform. It uses a derivative of Apache Tomcat as the servlet containerfor serving Web content, with an added component called Grizzly and uses Java New I/O forscalability and speed.The iRobot – The iRobot CreateTMVehicle, a mobile robot based on the Roomba platform (acommercial robotic vacuum cleaner). Unlike the Roomba, the iRobot Create was designedfor robotics development [1]. In place of the vacuum hardware, the iRobot Create includes acargo bay which houses a 25 pin port that can be used for digital and analog input and output.The Robajax Web User Interface – Is a website (http://www.robajax.com), where any usercan go to remotely control the iRobot. This website (hosted on godaddy.com) masksmaxwell.sdsu.edu, where the code for the UI based control resides.The Robajax Web Application Gateway – A SOAP based Web Service that acts a gatewaybetween the many browsers logged onto the Robajax Web User portal and the iRobotappliance.Maxwell.sdsu.edu – Is a server that hosts GlassFish and the JavaScript code for the RobajaxWeb User Interface.1.2

WHAT ISTELEROBOTICCONTROL?Telerobotic control is the area of robotics concerned with the control of robots from adistance, chiefly using wireless communications. Telerobotics is a sophisticated form ofteleoperation (doing work at a distance) [3] and combines two major subfields: teleoperationand telepresence (feeling like you are somewhere else). Two major components oftelerobotic control are the visual and control applications. A remote camera provides thevisual representation of the view from the robot. This remote camera will offer an operatorthe robot’s view, allowing intuitive control of the robot. Telerobotic control has not beenfruitful as time delay, resolution, and bandwidth required have only recently been adequate togive the ability to control a robot in any meaningful way. This is especially true in the case of

4ground-based control of space robots where the effects of time delays and bandwidthrestrictions are extremely significant [4]. In such a space system, the time delays (from theground to the space station) can be up to 3 seconds and this delay plays a major role in thestability control of the entire system. Bandwidth restrictions limit the availability of videoand audio channels. Therefore, less information passes between a user and the space-basedrobot.1.3

PREVIOUSRESEARCH

The field of telerobotics has been around for the past 40 years. Many researchendeavors have been dedicated to this field. Computer science and engineering curriculacommonly use the iRobot CreateTMVehicle (the iRobot) because of its low cost. The iRobotis particularly attractive because of the easy to learn Create Open Interface (OI) that uses theRS232 serial protocol for sending command packets and is the platform of choice tostimulate student interests and improve retention rates in many colleges. Weiss et al.identified that the iRobot as an ideal platform for researches seeking affordable and durablesolutions for teaching introductory robotic courses [5]. Wellman et al. at the University ofAlabama found that using the iRobot in Computer Science (CS) courses has helped boostenrollment and improve retention rates in the underrepresented minority groups and women[6]. Other research by Dodds in the Harvey Mudd College CS Department has found thatusing the iRobot positively stimulate students interest in CS and improves retention rates,especially in female students [7]. One of the more challenging areas students have focusedattention on is the integration of visual classification to automated robot task planning whereimage profiles consisting of pixel-intensity sums computed across subsets of a streamingvideo sample can be used for visual odometrical control of the iRobot, where the use ofimages taken from the iRobot's surroundings is used estimate change in position over time[8].Telerobotics is not just used to stimulate student interests but this field also has realworld applications as well. One of the most famous examples is the twin Mars ExplorationRovers (MER) project. The MER vehicles have a very sophisticated suite of autonomouscapabilities to explore unknown terrain and automatically collect data, such as scientificallyinteresting events [9]. The vehicles use hazards to populate a surrounding map of terrain

5traversability; from this map the best route is chosen to be executed. The vehicles repeat theprocess of detecting hazards and then executing the appropriate motion command until thegoal is reached. In order to maintain a local navigation map the vehicles combine wheelodometry, linear accelerations, angular velocities and optional vision-based pose estimates[10].Other research has focused on developing a system that combines both humaninteractions along with autonomous control, a concept called supervised autonomous robots[4]. In such a system there would a functional distribution of tasks between a user and therobot. The user would be responsible for: sensing or perceiving the work volume, reasoningand understanding the work volume, building a world model, generating a complete taskdescription, task analysis and understanding, task decomposition, error detection, anomalies,error recovery, and re-planning. The robot would then be responsible for collision freetrajectory generation and execution of run time functions and skills.Another use of telerobotics is on the Internet, focusing on the telesurgery applicationdomain. King et al. developed a protocol called Interoperable Teleoperation Protocol (ITP) tosolve the problem of using a shared data interface between telerobotic master-slave systems[11]. To simplify cooperation, ITP uses a low-overhead binary UDP packet structure.Telesurgery master-slave systems are connected with a master manipulator being operated bya surgeon in order to control the slave (patient-side robot).Many research projects have taken advantage of the Internet to develop highlyaccessible telerobotic systems. Hartfiel et al. designed a system to tackle issues ofsemiautonomous telerobotics, communication time delays, sensory feedback and assignmentand modification of tasks [12]. In Hartfiel’s research an electronic storyboard was the choseninterface because of the cooperative environments between humans and robots. The storyboard provided the following benefits: real time group problem solving, easy visibility of thesystem’s main functions, multiple operations can be performed simultaneously, and theinterface encouraged user interaction and ease of understanding. Zakaria et al. decided on anInternet based system to control their XR-4 series Robot Arm because of the manyadvantages the Internet had to offer: the Internet has extensive geographical reach, is networkand platform independent, is standards-based and open, and a wave of technologicaldevelopments, from high bandwidth networks to software technologies, is revolutionizing the

6Internet [13]. Zakaria’s Internet based system consisted of a robot, a camera and a personalcomputer. One of the biggest concerns was that the robot should not collide with otherobjects to avoid damage to both the robot and the objects. The user interface for thistelerobotic system was a button based browser with a live video stream embedded in thebrowser. To control the robot’s arms, a user would click buttons to perform a certaincommand while watching a video stream to see actions take place in real time.Using the Internet for teleoperation is not without challenges. The Internet facesproblems such as unreliable packet delivery and unpredictable latencies. Lakshmanan andRajkumar offered a solution to these problems in their research [14]. The main principle intheir research is to achieve reliable communication through redundancy, while maintaininglow end-to-end latency. They realized that individual links of the Internet can fail due tonetwork saturation or physical failures. Thus, they employed multipath routing techniques tosolve such problems. This was done by sending duplicate data to the same target throughdifferent paths. Thus, the network’s communication reliability increases with the degree ofstatistical independence between the chosen paths. Studies have been undertaken tocharacterize the correlation between paths [15]. Proactive re-transmissions were used to fixpacket errors due to tail-drop effects at router buffers and interference spikes in the wirelesschannel. The problem with waiting for acknowledgements before retransmitting data is thatthe resulting average packet latency is normally unacceptable. This was solved by pro-actively sending a duplicate copy of the data after a statistically estimated time delay. Thedelay was needed to accommodate the durations of burst traffic in the network. To achievereliable low-latency communication they used proxy forwarding, which also fixed problemscaused by an unreliable wireless link. Long latency in the Internet is mainly due to longdistances, which prevents acknowledgment based communication over the entire path. Aproxy node connected through the wired network could communicate over the Internet withthe controller with high reliability and without requiring acknowledgements.Another big field in telerobotics is autonomous mobile manipulating robots thatwould automatically perform desired tasks without human intervention. Xu et al. attached arobotic arm on the iRobot so it can be used to grasp objects [16]. The motivation behind Xu’swork was that it could be used to help people with motor impairments. Thus, the robotic armattached would allow the iRobot to grasp objects on the floor. Some of the challenges

7encountered were how to model such a robotic arm so that it can be used to grasp commonhousehold objects that come in different shapes and sizes. In this research, the robotic armwas modeled to emulate a human hand. To pick up an item people normally use their fingernail as a wedge to get under the item or fingers or the palm of the hand as a surface to pushan object and apply force to the opposite side to grasp the item. Thus, the robotic arm in thisresearch has a flat plate with a leading wedge to slide under an object, and a 2-link compliantfinger to apply force to grasp an object.Another research augmented the iRobot to enable service tasks that require spatialreasoning such as delivery, surveillance, and map maintenance [17]. This setup consists ofmounting an Axis207 Ethernet camera and six Sharp® GP2D12 Infrared sensors (IR) on topof the iRobot. The Wiimote game controller was used to direct the iRobot. Autonomousexploration was handled by a spatial-reasoning module. The software developed by Koziol etal. allowed a user to click on a goal location within the robot’s known map, and then thesystem automatically planned and executed a path to a target location.As mobile robots become more common, the need for robots to operate smoothly andcollision free arose. Research done by Snape et al. introduces a concept called hybridreciprocal velocity obstacles for collision avoidance among autonomous robots [18]. Thisconcept is an extension of reciprocal velocity obstacles [19]. Most avoidance algorithmsextrapolate where a moving object might be based on the object’s current velocity. However,treating another robot as a general obstacle using such a technique may not be the bestapproach because this technique does not take into account the reciprocity between robots(robots might react the same way to each other). Hybrid reciprocal velocity is a mix betweenreciprocal velocity obstacles and velocity obstacles [20]. Reciprocal velocity obstacles place50% of the responsibility on one robot (robot A) and 50% on the other (robot B) to avoid acollision, while for velocity obstacles it has one robot choosing a velocity that will avoid acollision. The hybrid formulation has a consequence that if robot A attempts to pass on thewrong side of robot B, full priority will be given to robot B to avoid collision. However, ifthe correct side is chosen, full cooperation with robot B is assumed and both robots retainequal priority.

8CHAPTER 2OVERVIEW OF THE SAN DIEGO STATEUNIVERSITY ROBAJAX WEB USER INTERFACEDEVELOPMENT PLATFORMIn order to start telerobotic control, a user will log on to the Robajax Web UserInterface (http://www.robajax.com) Web portal. The Robajax Web User Interface portal’smain page contains JavaScript code that creates the UI needed to control the iRobot. Whenthe UI detects that a user wants to move the iRobot, the UI will construct a SOAP Requestand use an instantiated XHR object to transmit a SOAP Request to GlassFish (a SOAPcontainer). GlassFish will then marshal the SOAP Request and invoke an operation on theRobajax Web Application Gateway Web Service to call the correct Web operation. TheRobajax Web Application Gateway will then construct a JSON object and send the serializedJSON object via a TCP socket connection to the iRobot to perform the requested command(see Figure 2.1).

Figure 2.1. Overview of the Robajax Web User Interface system.

92.1

MOTHERBOARD

The motherboard used for this thesis is a Migrus C787 DCF-P with a 1.2GHz EdenULV Processor [21, 22]. The view (see Figure 2.2) and specifications (see Table 2.1) of themotherboard are provided.

Figure 2.2. Picture of the Migrus motherboard.2.2

COMPACTFLASHMEMORY

CompactFlash (CF) is a mass storage device format used in portable electronicstorage. For storage, CompactFlash normally uses flash memory in a standardized enclosure[23]. For this telerobotic system a Kingston 4GB compact flash card is used (see Figure 2.3)and its specifications are provided in Table 2.2.2.3

DIGITALCAMERA

A Unibrain Fire-i™ Digital Camera is used in this telerobotic system [24]. It is alightweight camera suitable for mobile applications. The main features of this camera are two400Mbps Firewire ports (up to 640x480 resolution) and a maximum of 30 frames per second.A detailed specification of the camera is provided in Table 2.3.

The Robajax Web User Interface is an application developed using an AJAXmethodology. The AJAX methodology represents a group of technologies that can be used tocreate a web application that communicates with a server without having to interfere with thecurrent stage of the web application. Jesse James Garrett explained the followingtechnologies that can be used [2]:1. HTML or eXtensible HyperText Markup Languague (XHTML) and Cascading SytleSheets (CSS) for presentation2. Document Object Model (DOM) for dynamic display of and interaction with data3. XML for the interchange of data, and Extensible Stylesheet LanguageTransformations (XLST) for its manipulation4. XHR object for asynchronous communication5. JavaScript for bringing these technologies togetherDevelopment and user experience would differ drastically if developed understandard HTML forms [25]. Using HTML forms the UI would contain buttons. Pressing abutton would generate either a POST request method or a GET method that would cause datato be sent to the server to be handled, which in turn would cause the iRobot to perform therequested action. Afterwards, the server would then respond by sending back a complete webpage, triggering the browser to re-render the website. Clicking buttons at a fast rate(navigation through a series of obstacles) would cause many page refreshes. This wouldsaturate the server causing communication between the server and the web page to becomevery slow. This is due to the increased communication time due to continuous sending POSTrequests and re-rendering the browser, which would make it very distracting to a user whiletrying to watch the video stream. In order to stream live videos, HTML frames would have tobe used (one to control the iRobot and one to stream live video) [26].Using an AJAX methodology, the web page rendering a user interface would neverneed to be refreshed. Server responses are no longer full HTML web pages but rather smallmessages in the form of XML. The responses are sent asynchronously using XHR, resultingin a much faster response time due to the communication model being asynchronous and thedata exchange being much smaller than the HTML forms counterpart. A fast response time isalways desired as it is a key component to all UI based controls.

15With AJAX, XHR objects are used to invoke Web Services asynchronously. This isachieved because XHR objects are invoking Web Services and waiting for a response in thebrowser’s background process. The invocation of a Web Service is done by sending a SOAPRequest message and waiting for a SOAP Response message. Since this communicationhappens in the browser’s background process and decoupled from the main browser eventloop, data can still be sent and received while a user is performing any main browser action.With a traditional HTML forms method, a user would have to click a button to submit therequest and wait for the HTML form to be reloaded before continuing. However, during thisreload process the browser is blocked prevent any action from taking place.Another benefit offered by AJAX is reduced network traffic on the server. With aHTML forms model every browser connected to the Robajax Web User Interface wouldconnect to the server and the server would have to send a complete HTML web page to everybrowser. Imagine the scenario where every single browser begins to direct the iRobot tomove, the server would again have to send a complete HTML web page to every browser.Using an AJAX methodology, the JavaScript code on the browser is responsible forrendering the web page. The server now has a smaller job because now the server only has torespond with small messages. This reduced network traffic on the server allows the RobajaxWeb User Interface to scale very well, as the added network traffic between one browsersession connected to the server versus many browser sessions is very minimal. Because ofthese advantages, using an AJAX based methodology is a superior technology to use forWeb-based for telerobotic control system interfaces.Ease of access is important to all UI based controls. If the Robajax Web UserInterface were to be developed as a monolithic personal computer (PC) application, theRobajax Web User Interface would only be accessible to users with a PC. Any UI basedcontrol wants to expand its potential list of users to as many users as possible. Being a webpage, the Robajax Web User Interface is readily available to anyone with Internet access.With net books and smart phones rising in popularity, the Robajax Web User Interface willhave more and more potential users. Devices ranging from PCs to personal digital assistants(PDAs) to an Android Phone can logon to the Robajax Web User Interface and control theiRobot.

162.5

THEROBAJAXWEBAPPLICATIONGATEWAY

The Robajax Web Application Gateway is a Web Service designed to reduce theamount of network connections (sessions) to the iRobot. When the iRobot boots up, theiRobot will connect to the nearest access point. The iRobot will then send a broadcast requestlooking for a Dynamic Host Configuration Protocol (DHCP) server to answer. The routerwill direct the request to the correct DHCP server and that DHCP server will then assign anInternet Protocol (IP) address to the iRobot. Afterwards the iRobot will establish a TCPsocket connection with the Robajax Web Application Gateway. Suppose there are multipleusers on multiple networks that want to control the iRobot. Each user would open a webbrowser and log on to the Robajax Web User Interface to interact with the UI. Every SOAPRequest constructed by the UI will be transmitted to the Robajax Web Application Gateway;the Robajax Web Application Gateway will then invoke a web operation to create the correctJSON message to transmit to the iRobot. Thus, the Robajax Web Application Gateway canhave many sessions, but the iRobot will have only one session (see Figure 2.4).

17The main purpose of restricting the number of sessions over the WLAN to one is toprevent the streaming of redundant video to multiple parties, thus allowing the wirelesschannel to carry a single video stream to the Robajax Web Application Gateway with thehighest frames/sec or lowest QScale value.Without the Robajax Web Application Gateway, communication between the iRobotand the UI would look vastly different. One possible way is to store the IP address assignedby the DHCP server as a Fully Qualified Domain Name (FQDN). With this implementation,after the DHCP server assigns an IP to the iRobot, it would communicate this information tothe DNS on gammabot.sdsu.edu and the DNS will store the IP address would be stored as aFQDN. Now every SOAP Request constructed by the JavaScript code by the UI will betransmitted to a GlassFish server on the iRobot (see Figure 2.5), while any status informationfrom the iRobot would be transmitted to each user as well. With the same scenario as above(multiple user opening a browsers), this would multiply the amount of sessions on the iRobotas a function of the number of users. As stated above, sessions maintained by the iRobotshould be minimized. Thus, this second implementation without using the Robajax WebApplication Gateway is not an ideal solution because it does not allow this telerobotic systemto be scalable.2.6

GLASSFISH

Since this system architecture uses a Web Service (the Robajax Web ApplicationGateway) to decrease the amount of network connections to the iRobot, there were twochoices between which protocol to use for exchanging information, SOAP or REST. Becausethere is not a standard for RESTful Web Services, SOAP was the chosen Web Service.Another benefit of using SOAP is that there already exists a powerful SOAP container(GlassFish) that comes packaged with NetBeans, the integrated development environment(IDE) used for this thesis.2.7

THE IROBOT

The iRobot CreateTMVehicle (the iRobot) consists of electronic and softwareinterfaces for controlling the iRobot’s behavior and reading sensors (see Figure 2.6). Theelectronic interface includes a 7 pin Mini-DIN connector and a DB-25 connector in the cargo

18

Figure 2.5. The iRobot will generate more traffic if the Robajax Web ApplicationGateway is not used.

Figure 2.6. The iRobot CreateTMVehicle.

19

bay for connecting hardware and electronics for sensors. The software interface allowsmanipulation of the iRobot’s behavior and reading sensor data through a series of commands.Because of limitations in storage space and processing power of the iRobot, theaforementioned motherboard (Refer to Section 2.1) was attached to the iRobot. The iRobotCreateTMVehicle was chosen because of the affordability and flexibility in extending themodel’s functions.

20CHAPTER 3USER INTERFACE DESIGNThe challenge of every telerobotic system is to create a user interface that is bothergonomically friendly and easy to use. Many UIs were developed during this research effort.The many revisions of the UI, ranging from the beginning button based UI to the finalimplementation of the circle based UI, are discussed in this chapter.3.1

BUTTONBASEDUIThe button based UI (see Figure 3.1) was the very first design implemented to controlthe iRobot platform. Its design allowed the user to control the iRobot in a very simplistic andeasy to understand manner. The Forward/Backward buttons allowed the user to move at aconstant speed of 100 mm/s, either forward or backwards, while the left/right buttonsallowed the user to move at a speed of 100 mm/s while turning at a radius of 100 mm. Theradius is measured from the center of the turning circle to the center of the iRobot; the longerthe radius the straighter the iRobot will drive before turning (see Figure 3.2) [1]. A stopbutton was used to order the iRobot to halt. However, during test runs the following issueswere discovered:1. Lack of speed control: Because of this UI’s simplistic design, the user wasn't able tocontrol how fast they wanted the iRobot to move. This UI only allowed the user tomove the robot at a constant speed of 100 mm/s. This created various issues if theuser wanted to have the iRobot move faster, a likely scenario if the user wanted theiRobot to cover a vast area quickly.2. Inability to control wheel angular velocity and turning radius: Similar to the lack ofspeed control, this design didn't offer the user an ability to control the turning speedof the iRobot. The iRobot was only able to be controlled to move at constant speedand turning radius. This deficiency prevented the iRobot from being able to performsharp turns, which is problematic if there are a lot of obstacles in the iRobot's area.3. Cumbersome to use: Even though the button based GUI was easy to use andunderstand, the amount of buttons made it an encumbrance. While controlling theiRobot, if any turn commands were needed, the user had to click the left/right buttonrepetitively. This is especially frustrating to a user if there are a lot of obstacles in theiRobot's area. With the amount of buttons needed to control the iRobot, users can

21

Figure 3.1. Screenshot of the button based UI.quickly develop hand and finger fatigue. This would cause this telerobotic system tonot be non-ergonomic.The video stream’s low resolution is caused by a high QScale. High QScale reducesthe quality of the video stream, inversely; a low QScale will increase quality. High frame rateallows fast video streaming, whereas a low frame rate will make the video stream appearchoppy to a user. The best video stream would consist of a low QScale and a high frame ratefor fast streaming high quality video. These two values, QScale and Frame/s, must combineto stay within in the Encoding Bitrate [27].Depending on the scenario, the QScale and Frame/s can be varied. For example, whendriving the iRobot, emphasis should be placed on a fast video stream but not quality. So, boththe QScale and Frame/s should be high. However, when a remote user wants to look at anobject in front of the iRobot, emphasis is placed on quality and not how fast the frame rate is,

22

Figure 3.2. How the iRobot turns based on the radius.because it is not useful to see a fast stream if it is hard to determine what object is in front ofthe iRobot. Thus, in this scenario, both the QScale and frame rate should be low.While this design served as a good starting point as a GUI to control the iRobot, thebutton based UI contained a lot of key problems that needed to be addressed (see Table 3.1)to make it ergonomically user-friendly.Table 3.1. Button Based UI SummaryAdvantages DisadvantagesEasy to use/understandLack of Speed ControlLack of Turn Speed/RadiusCumbersome to use

3.2

SLIDERBASEDUIThe Slider based UI (see Figure 3.3) was the second design. It was created to addressthe issue of not being able to control the speed of the iRobot. The slider in conjunction withdirectional buttons allowed a user to better control the motion of the iRobot. The slider

23

Figure 3.3. Screenshot of the slider based UI.controlled the straight-line speed of the iRobot; it allowed the user to set the speed of theiRobot up to 0.5 m/s (500 mm/s).Clicking the left and right button made the robot turn at a radius of 100 mm and moveforward at a speed of 100 mm/s. However, once the mouse button was released, anothercommand was issued, causing the iRobot to move forward at the previous speed requested.For example, say the slider was positioned at 0.25 m/s and then the user clicked the rightbutton. The iRobot performed the requested turn at a speed of 0.1 m/s. Once the user releasedthe mouse button, the iRobot continued moving straight ahead at a speed of 0.25 m/s.Holding the left and right button caused the iRobot to turn for a longer time period beforetraveled straight at the previous speed. Thus, the iRobot can be issued to move in a circle.Even though, this design resulted in a greater control of the iRobot, test runs revealedthe following issues:1. Lack of Turn Speed/Radius: Since the UI features the same left/right button as theButton GUI (refer to section 3.1), this problem wasn’t fixed, limiting the user in thatthe robot could only be made to turn at a constant velocity.2. Inefficient Left/Right maneuvering: Once the user clicked the left/right button, aSOAP Request is sent to GlassFish on maxwell.sdsu.edu, which then creates a JSONobject and passes the JSON object along to gammabot.sdsu.edu to perform therequested turn command. After the mouse up event (when a user lets go of a mousebutton), another SOAP Request is sent to command the iRobot to move forward at themost recent slider invoked speed, as explained earlier. This is inefficient as it makesboth maxwell.sdsu.edu and gammabot.sdsu.edu process two requests. Thus every turn

24command caused the iRobot to perform two motion requests. If a user is navigatingthe iRobot through a series of several obstacles, both servers can be inundated withmessages that will saturate the network.3. Hard to Stop Using Slider: While the slider allows the user to easily control the speedof the iRobot, it was hard to set the slider at an exact speed. Normally being a coupleof millimeters off did not pose a big problem. However, trying to adjust the slidervalue to be exactly 0mm/s to force the iRobot to stop was challenging. This isworrisome as there are cases where the iRobot needed to stop immediately to preventthe iRobot from colliding with a rigid body or driving off a ledge. This problem wasfixed by using both the stop button in conjunction with the slider. However, this sliderbased UI is an inefficient design to force the user to use two different mechanisms forcontrolling the same feature.4. Cumbersome to Use: While controlling the iRobot, a user had to move the slider tomake the iRobot move forward, click the turn buttons to make the iRobot adjust to theleft or the right, and use the stop button to halt the iRobot. All these UI sequencesrequired the user to perform a lot of mouse clicking and mouse movement whichcould be exhausting for a user.Once again, another design was need to address these issues (see Table 3.2).Table 3.2. Slider Based UI SummaryAdvantages DisadvantagesAble to control the speedLack of Turn Speed/RadiusInefficient Left/Right maneuveringHard to Stop Using SliderCumbersome to use3.3

CIRCLEBASEDUIThe Circle based UI (see Figure 3.4) was the third and final design. Issues thatplagued the first two designs, namely the inability to vary the turning velocity and radiuswere solved. In the Circle based UI the user controls the iRobot by maneuvering the mousewithin a circle. The circle has a top view point of reference. The mouse position within thecircle is polled every second by the JavaScript code, from that mouse position a vector isdrawn using the center as the initial point. This creates two key attributes (1) robot velocity isproportional to the magnitude of the vector; (2) orientation (turning radius) is a function oftheta, the angle of the vector. So the position in the circle completely defines the motion ofthe iRobot.

25

Figure 3.4. Screenshot of the circle based UI.This design can be best thought of as if you are driving a car. If the vector is a straightline forward, the iRobot will go straight. If the vector is a straight line backward, the iRobotwill drive in reverse. If there’s a slight angle, then the iRobot will turn its wheel and move inthe direction specified. Like making a turn with a car, if you do not straighten out the wheels,the iRobot will continue to drive in a circle.The mouse position is polled every second, if there is a significant change (refer toSection 5.1) since the previous mouse position, the iRobot will be told of the new vector tomove (see Figure 3.5). Moving the mouse over the red dot (see Figure 3.6) will cause theiRobot to immediately stop by sending an immediate stop message, even if the polling periodhas not matured (Tmature) as in indicated in Figure 3.6. If the user decides that polling everysecond is not sufficient, the user can click within the circle. This will cause the iRobot toimmediately perform the requested movement command. Since the vector can only bedetermined within the circle, moving the mouse outside of the circle will cause the iRobot toimmediately stop.

26

Figure 3.5. A decision diagram for tracking mousemovement within a web browser.

Figure 3.6. A SOAP message is sent immediately when the mouse enters thered circle.

27The Circle based UI design addresses every previous issue encountered. The user canstill control the speed of the iRobot with the added benefit of controlling the turning radius,thus it is more efficient as the request to the iRobot is now a single command (see Table 3.3).Stopping is now trivial as once the mouse moves over the red dot, the iRobot willimmediately stop. A user does not have to use a mixture of user interface widgets (sliders andleft/right buttons) to control robotic motion. In addition, a user does not have to continuouslyclick a mouse button, so it is ergonomically friendlier. Controlling the iRobot using the circlebased UI is similar to driving a car, so it is easy to understand as it is based on a conceptmany users are familiar with.Table 3.3. Circle Based UI SummaryAdvantages DisadvantagesControls speed/turn radius in 1 commandEasy to stop the iRobot motionErgonomically friendlyMouse clicking is optionalEasy to useOne second polling might not fit user's needsMouse clicking might still be needed

The only issue encountered is the mouse polling period of one second. It is difficult toguess what the user’s needs are in advance, so the default period is configured to be onesecond. However, the mouse click feature was added in case the user cannot wait for anentire second.

28CHAPTER 4PROBLEMS ENCOUNTEREDThe Spiral software development model [28] was used to develop the iRobot’s userinterface. During the software testing phase many issues were encountered. These issueswere fixed in subsequent iterations. The following problems encountered and a discussion ofthe methods used to fix them is presented in this chapter.4.1

INCORRECTSTATUSBEINGDISPLAYED

IRobot provides certain status information that would be useful for a remote user.Status such as battery capacity, battery temperature, current charge state, voltage and moreare all pertinent information when remotely controlling a remote robot. The user would be ina potentially dangerous predicament is a remote robot is not responsive because of a deadbattery without being issued any type of information for them to monitor.However, status information coming back at one stage in the development processwas nonsensical. Charge states of unknown value when the iRobot was charging and batterytemperature readings of 114 degrees Celsius (impossible as that battery would be burning)were being returned initially. It was thought that the Java Communications 3.0 API was theproblem because it could not provide accurate ways of reading data [29]. So many solutionswere tried, solutions varying from using the example code of reading data from the serial port(by applying an event handler) to manually reading the serial port, byte by byte. These initialstrategies did not work.Finally, it was determined that the MAX232 chip designed to connect to the iRobot’sserial port was wired wrong. The error was fixed and status information is now coming backcorrectly.4.2

WEBSITEPICTURES NOTBEINGLOADED

After the slider based UI (refer to Section 3.2) was developed it was noted that theslider picture was not being loaded correctly. Even though the static html code was written

29correctly, the picture would only load correctly in the Internet Explorer but not the FireFoxbrowser.This problem was inadvertently solved using AJAX methodology. The static htmlcode was being changed to be dynamically created in JavaScript when the website loaded.Now the pictures load correctly in either FireFox or Internet Explorer.4.3

OUT OFORDERSOAPMESSAGES

During test runs it was noted that SOAP messages coming from the Robajax WebUser Interface were sometimes arriving out of order. At the time, every SOAP messagereceived by the Robajax Web Application Gateway was blindly processed in the orderreceived. This caused problems because the out of order SOAP messages caused the iRobotto move erratically. For example, with the slider based UI, clicking the left or right buttonsends two SOAP messages to the Robajax Web Application Gateway. The first SOAPmessage tells the iRobot to turn, the second SOAP message (which the Robajax Web UserInterface sends later) tells the iRobot to drive straight. However, sometimes the Robajax WebApplication Gateway receives the second SOAP message before the first SOAP message (seeFigure 4.1). This causes the iRobot to continually turn in a circle until another motioncommand is sent in.

Figure 4.1. SOAP request messages being processed out of order.

30To alleviate such problem, a message queue was implemented. SOAP messagescoming from the Robajax Web User Interface would have a sequence ID attached. When auser first logs onto the Robajax Web User Interface UI page, a sequence ID (0-65535) wouldbe randomly generated. The Robajax Web User Interface will then establish a handshakewith the Robajax Web Application Gateway to send the sequence ID. The sequence ID isused to make sure SOAP messages are processed in the order they were generated, not theorder the Robajax Web Application Gateway received them (see Figure 4.2).

Figure 4.2. SOAP messages being processed by in order by using sequence ids.SOAP messages received by the Robajax Web Application Gateway would be storedin the priority queue. If the messages are arriving in sequential order, the messages wouldimmediately be processed. However, if messages are arriving out of order, then the queuewould wait until a certain timeout interval has expired and then execute the out of ordermessages (see Figure 4.3). The timeout interval was put in place in hope that the correctSOAP message would be sent during that time and re-synchronize the sequence ID.

31

Figure 4.3. New expected sequence ID due to an out of order SOAP message.4.4

SDSU

INTERNALWLAN

FIREWALL

The initial design had the iRobot act as a server; taking incoming JSON messagesfrom a client, the Robajax Web Application Gateway (see Figure 4.4). However, during fallof 2009 an internal WLAN firewall was erected by SDSU’s IT department that rendered suchoperation invalid. This firewall blocked any programs on maxwell.sdsu.edu from establishingan initial connection to devices on the wireless LAN (see Figure 4.5), essentially forcing anyprogram on maxwell.sdsu.edu to act as a client. Thus, the firewall allows TCP sessions to beestablished if the session originator is a device on the wireless network. This created aproblem because the Robajax Web Application Gateway and the iRobot could no longercommunicate with each other.

32

Figure 4.4. Connection initiator before implementation of the SDSU firewall.

33The only fix available was to make the Robajax Web Application Gateway a serverand have the iRobot initiate the connection (see Figure 4.6). The iRobot and the RobajaxWeb Application Gateway initially used SOAP messages to communicate with another.During this time, it was decided that SOAP messages were too data intensive to use for suchtrivial task like directing the iRobot where to move. Instead of using SOAP messages, theiRobot and the Robajax Web Application Gateway would now communicate using light-weight JSON messages through a TCP socket.

Figure 4.6. The iRobot is now the connection initiator to adapt to SDSU firewallrestrictions.The GlassFish sever on the iRobot was removed because the Robajax WebApplication Gateway now only uses JSON messages to communicate with the iRobot.Because of this change, messages were being processed at a much faster rate and delayperiods noted during earlier iterations from using GlassFish and SOAP were negligible.Initially it was thought that UDP sockets would be ideal for this re-architecture,because UDP is a lighter weight transport protocol and the extra connection oriented featuresof TCP were not needed. A few lost motion messages would not be detrimental to the iRobotbecause a user can easily perform the same actions to have those messages sent again.Additionally, the overhead to maintain a TCP socket was not needed. Unfortunately, the

34SDSU firewall seems to block UDP protocols between the wired and wireless LAN, so TCPhad to be used instead.4.5

LOSS OFWIRELESSSIGNAL

Successful transmission between the iRobot and the Robajax Web ApplicationGateway is highly coupled with the wireless signal strength. During test runs it was notedthat occasionally, in areas of low signal strengths or areas covered by a lot of metalinfrastructure, some messages sent to the iRobot were never received. Problematic areas tendto be regions containing a lot of metal infrastructure and farther away from the access point.Thus, a remote user would sometimes be required to manually recover the iRobot from anarea of poor signal strength. Using debugging tools it was noted that this problem started tobecome more frequent as the wireless bit rate got closer to 1 Mb/s. Thus, a mechanism wasneeded to determine when the iRobot and the Robajax Web Application Gateway are nolonger connected, so the connection could be restarted. This problem was solved by using a“heartbeat” message.The “heartbeat” refers to a message, requesting the iRobot’s status (see Figure 4.7),that the Robajax Web Application Gateway will send to the iRobot every 3 seconds (seeFigure 4.8). IRobot has 5 seconds to respond to the signal; otherwise the Robajax WebApplication Gateway will close the existing TCP socket connection, open a new connectionand wait for the iRobot to re-establish connection (see Figure 4.9). The delay period wasneeded to give ample time for the JSON response message to be returned. 5 seconds waschosen because test runs showed such a delay provided ample time for a JSON responsemessage to be returned. A user will know when this has happened, because the status “Noconnection to the iRobot” will be displayed on Robajax Web User Interface. Ideally, therewould be some native Java library that can be used to tell if there is still an existingconnection between the iRobot and the Robajax Web Application Gateway, because theresponse might just take a long time because of low connectivity. However, we were not ableto find such a library. So, the only solution is to assume the iRobot is either down or has lostconnection to the Robajax Web Application Gateway.

Figure 4.7. Sample GetStatus JSON message.

35

Figure 4.8. Heartbeat between theWriteThread and the iRobot is still alive.

Figure 4.9. Connection restart due to a lost heartbeat message.

36On the iRobot’s side, it expects the “heartbeat” to be sent by the Robajax WebApplication Gateway every 5 seconds. After the time elapse, it will assume communicationto the Robajax Web Application Gateway is lost and continually try to re-establish aconnection. This autonomous solution allows the iRobot to function in areas of lowconnectivity with no user intervention. It should be noted that areas with very lowconnectivity will quickly drain the iRobot’s battery at a faster rate as it continually trying tore-establish connection.4.6

THEROBAJAXWEBAPPLICATIONGATEWAYDEADLOCKCONDITION

The use of TCP socket along with a “heartbeat” to monitor the health of theconnection worked well during trial runs. However, leaving the connection open for anextended amount of time (overnight) would cause the Robajax Web Application Gatewayand the iRobot to lose their established connection. As a side effect the Robajax WebApplication Gateway would also enter a dead lock scenario (see Figure 4.10).

Figure 4.10. Deadlock scenario caused by a lost TCP connection.

37The problem occurred when the Robajax Web Application Gateway sent a“heartbeat” message to the iRobot (thus entering a blocking Input Output (IO) state), whomduring this time just happens to lose the established TCP socket connection. Because theiRobot has just lost the connection, a response message cannot be sent back to the RobajaxWeb Application Gateway. Complicating matters is that Java’s native socket interface has noway of determining when an established socket connection has been lost. Thus, the RobajaxWeb Application Gateway will continue to be stuck in its blocking IO state, waiting for anIO response that will never happen.This problem was fixed by using an updated socket interface, SocketChannel [30].SocketChannel provides the ability to interrupt a blocking IO call, allowing control to beregained if it is determined that the established socket connection is no longer viable. Now,every time the Robajax Web Application Gateway sends any message (like the “heartbeat”)to the iRobot, a timer is started. If the iRobot does not respond in 5 seconds, the timer willwake up and interrupt the blocking IO call. The Robajax Web Application Gateway will thenclose any existing socket connections and open another one waiting for the iRobot to re-establish the connection.On the iRobot, a timer is also used. If the iRobot does not receive a message from theRobajax Web Application Gateway in 5 seconds, the message’s timer wakes up and interruptthe blocking IO. The iRobot will then close off any existing socket connection, issue a stopcommand (if the iRobot is not on the home base, a precaution taken just in case the iRobot ismoving) and attempt to re-establish a connection with the Robajax Web ApplicationGateway after a wait period. The 5 second wait period is needed because when the iRobot’swireless connection is dropped, the iRobot needs some time to re-initialize the wireless link.Attempting to establish a connection with the Robajax Web Application Gateway before thelink has been re-initialized will cause ICMP network unreachable errors.The reason why the timer wakes up in 5 seconds is because the heartbeat is set to betransmitted every 3 seconds, and 2 seconds is added on top of that to give some lag time fortransferring messages across a socket. Since this on average only allows enough time for onemessage to be through, all messages sent between the Robajax Web Application Gatewayand the iRobot must be accounted for. The rationale being, if the wireless connection is

38dropped, we do not want the iRobot to continue moving as there would not be a way to stopit remotely. The risk is too great for the iRobot to collide with a rigid object or drive off aledge. It is much safer to have both the Robajax Web Application Gateway and the iRobotstop its current task and re-initialize the connection to get back to a known state.

39CHAPTER 5TELEROBOTIC DESIGNThis chapter talks about the overall design of this telerobotic system. Designing atelerobotic system with these ideas will make a telerobotic system scalable, portable,ergonomically friendly, and accessible.5.1

CIRCLEUI

DESIGN

This section talks about the design of the Circle UI. The main focus of the Circle UIis to make the UI ergonomically friendly and accessible.5.1.1 AlgorithmWhen a user logs on to Robajax Web User Interface, the very first thing that needs tobe done is synchronize the sequence ID with the Robajax Web Application Gateway (seeFigure 5.1). This sequence ID is needed to ensure SOAP Requests are processed by theRobajax Web Application Gateway in the order they were submitted by a user through abrowser (Refer to Section 4.3). Afterwards, the two timers are started; the first timer is usedto poll the Robajax Web Application Gateway for the iRobot’s status. The status is importantbecause the status contains relevant information (battery temperature, battery charge, etc) auser might need to determine the next best course of action. For example, by monitoring thebattery charge, a user could best determine when is the best time to direct the iRobot to returnto the home base to recharge the battery. An unresponsive the iRobot due to an empty batteryis a high nuisance to a user, as they would have to manually retrieve and transport the iRobotback to the home base to re-charge, which would defeat the purpose of telerobotic control.It is imperative that every mouse movement that occurs within the circle based UI isnot translated as a motion command, causing JavaScript to create a SOAP message and sendthe SOAP message to the Robajax Web Application Gateway (see Figure 5.2). This isbecause when dealing with any hardware, like the iRobot, requests that cause physicalmotion need time to be processed. Otherwise, both servers (the iRobot and GlassFish running

40

Figure 5.1. A UI handshake sequence diagram.

Figure 5.2. A UI send command sequence diagram.

41on the gateway server) will become saturated and can malfunction. Thus, mouse movementswithin the circle will be polled every second (a time period chosen because it offered justenough time to not saturate the Robajax Web Application Gateway and the delay did notcause complaints during demonstrations with inexperienced users) to give the iRobot enoughtime to process any request it receives. Even though, mouse movement within the circlebased UI is polled every second, this does not mean a SOAP message is issued every second.Issuing the SOAP messages every second when the mouse has not moved is not efficient. Tofurther help reduce the amount of data transmitted over the network, SOAP messages willonly be sent if the current mouse position is greater than or equal a Euclidean distance equalto three pixels than the previous mouse position used for the most recent SOAP message.Test runs have shown that three pixels yielded acceptable results of not saturating thenetwork. During test runs, users did not even notice this restriction, because most of the timethey are moving the mouse beyond this arbitrary movement threshold.Sometimes a user needs the iRobot to respond immediately and they cannot tolerate a1 second delay period or the 3 Euclidean distance threshold. To alleviate such issue, if a userclicks within the circle, a SOAP message (see Figure 5.3 for a sample SOAP message) isimmediately created and sent to the Robajax Web Application Gateway. Stop commands(moving inside the stop region or outside the circle) also have the same effect. It is a bad ideato enforce a delay period before stop commands because the iRobot could possibly collidewith a rigid object or drive off a ledge during such delay period.

Figure 5.3. Sample of a SOAP request message.

425.1.2 Static Class DiagramTo be consistent with object oriented programming (OOP), the following classes (seeTable 5.1) were created as part of this thesis work (see Figure 5.4).Table 5.1. Circle UI’s Static ClassesClass Name High Level DescriptionMainApplication class that coordinates the rendering of all UI controls.This class also tracks mouse movement and forwards suchmovement to the circle control where the movement will eitherprocessed as a motion command or ignored if the mousemovement is outside of the circle control.IRobotCommunication class used to send SOAP messages to the RobajaxWeb Application Gateway and process responses returned.CircleUtility class used to render the circle user interface. This class alsodetermines if mouse movements should be processed as a motioncommand (if the movement is within the circle UI rendered).String Utility class used for manipulating strings.PictureUtility class used to create dynamic html code for insertingpictures.DialUtility class used to render the dial controls; the controls monitorthe iRobot's speed and direction.

5.2

THEROBAJAXWEBAPPLICATIONGATEWAYDESIGN

This section discusses the design of the Robajax Web Application Gateway. Themain focus of the Robajax Web Application Gateway is to make this telerobotic systemscalable in terms of the number of supported remote users and remote viewers.5.2.1 AlgorithmThe Robajax Web Application Gateway is used as an intermediate service pointbetween the Robajax Web User Interface and the iRobot. By design, the iRobot has adynamic IP address because this increases the portability of this Telerobotic system. With adynamic IP address the iRobot can be placed in any wireless network, receive an IP address(through a DHCP server) and communicate with the Robajax Web Application Gateway. If adynamic IP address is not used, a user would have to configure a subnet mask and

43

Figure 5.4. The circle UI’s static class diagram.

44a default gateway address and one or more name server addresses for every wireless networkthe iRobot wants to use. Since the iRobot has a dynamic IP address, to communicate withRobajax, an intermediate gateway with a static IP address is needed.Since, SDSU’s firewall prevents the Robajax Web Application Gateway frominitiating a connection to devices on the campus’ wired network, the Robajax WebApplication Gateway is forced to act as a server. The Robajax Web Application Gateway’sfirst course of action is use Transmission Control Protocol (TCP) to open a socket (SDSU’sfirewall prevents User Datagram Protocol (UDP) from being used) and wait for the iRobot toestablish a connection (see Figure 4.6 and Figure 5.5). During this non-connected state, anySOAP messages sent by the Robajax Web User Interface are not processed by the RobajaxWeb Application Gateway.

Figure 5.5. The Robajax Web Application Gateway start server sequence diagram.To prevent unknown programs from establishing a connection with the Robajax WebApplication Gateway, the session needs to be authenticated with a shared secret key. So, theiRobot would first establish a connection with the Robajax Web Application Gateway andthen send the shared secret key to authenticate the session. Once the correct password hasbeen sent, the Robajax Web Application Gateway will transition to a ready connection state

45(NOT_READY → READY) and start to process SOAP messages sent by a user through theAJAX application running within a user’s browser. Since it is important to know the status ofthe iRobot, every 3 seconds, the Robajax Web Application Gateway will send a JSONmessage to request for the iRobot’s status. This periodic polling is also used to determine ifthere is a valid connection with the iRobot. An invalid connection will result in the TCPSocket being closed and re-opened (see Figure 5.6).

Figure 5.6. The Robajax Web Application Gateway close server sequence diagram.A sequence ID, arg0, of the SOAP message (see Figure 5.7 for a sample handshakeSOAP message) sent by the Robajax Web User Interface to the Robajax Web ApplicationGateway, a process known as handshaking (see Figure 5.8), to ensure commands areprocessed in order. These sequence ids ranges from [0-65535] and are always sequential(except when wrapping from 65535 to 0). SOAP messages sent to the Robajax WebApplication Gateway are stored in a priority queue. Messages in the queue that arrived inorder are processed immediately. Messages that arrive out of order incur a 2 second penaltybefore the message is processed; with the hope of the correct message being sent during thatdelay period to re-synchronize the message order. If the correct messages do not arrive, theout of order messages are processed (see Figure 4.3). Furthermore, the priority queue is

46

Figure 5.7. Sample of a handshake SOAP request message.

Figure 5.8. The Robajax Web Application Gatewayhandshake sequence diagram.designed to ignore any messages with a smaller sequence ID than the most recent messageprocessed (with the exception of the wrap around case).Since motion commands to the iRobot causes the iRobot to move, it is important toensure that messages sent to the iRobot are not dropped, due to network saturation or themessage queue being full, (see Figure 5.9). This is because if a user notices that the iRobot isabout to collide with a rigid object or drive off a ledge, they would want to divert the iRobotaway from danger immediately. Thus, every command sent to the iRobot is attached to atimer (see Figure 5.10). If the iRobot does not respond before the timer matures (5 seconds),the Robajax Web Application Gateway will transition to a non-connected state, close off anyexisting connections and wait for the iRobot to establish another connection. The value forthe timer has to be between the heartbeat timer (3 seconds) and an ample time when aresponse is expected. Test runs have shown that 5 seconds yielded acceptable results.

Figure 5.10. The Robajax Web Application Gateway write task sequence diagram.5.2.2 Static Class DiagramTo be consistent with object oriented programming (OOP), the following classes (seeTable 5.2) were created as part of this thesis work (see Figure 5.11).5.3

THE IROBOTDESIGN

This section discusses the design of the iRobot user interface. The main focus ofdesigning the iRobot UI is to make the interface portable.

48Table 5.2. The Robajax Web Application Gateway’s Static ClassesClass Name High Level DescriptionGatewayIs a Web Service that contains web operations which areinvoked when a SOAP message intended for the RobajaxWeb Application Gateway is received.WriteThreadA thread that creates and maintains the connectionbetween the Robajax Web Application Gateway and theiRobot. This thread determines when to process a SOAPRequest the Robajax Web Application Gateway receivesby storing it in a queue and handles sending/receivingJSON messages from the iRobot.ConnectionStateA class used to store the connection state between theRobajax Web Application Gateway and the iRobot.SoapRequestA container for the SOAP Request received. When aSOAP Request is received, it will be bundled with a timestamp. This time stamp allows the WriteThread todetermine when to process the SOAP Request.WriteTaskA class dedicated to determine the status of a JSONmessage sent by the Robajax Web Application Gateway tothe iRobot. This status is used to determine whether theconnection between the iRobot and the Robajax WebApplication Gateway is still valid.ParserA class used to parse the IP address of the Robajax WebApplication Gateway.

495.3.1 AlgorithmThe very first action item for the iRobot is to open a serial port on the iRobot (seeFigure 5.12). Because of the importance of the iRobot’s status information, the iRobot willperiodically send a request for the iRobot’s status to the iRobot’s Micro Controller Unit(MCU) over the serial port every 3 seconds. This is done frequently and autonomouslybecause the iRobot’s status is a very important piece of information that needs to berefreshed constantly regardless of whether there is a user logged on to the Robajax Web UserInterface or not. This is done for 2 reasons: (1) Since the Robajax Web Application Gatewayalways knows the status when a user logs onto to Robajax, the status can immediately bedisplayed and (2) the Robajax Web Application Gateway automatically logs the status in ahistory file allowing for diagnostic post processing and analysis.

Figure 5.12. The iRobot open port sequence diagram.Afterwards, the iRobot will then need to establish a wireless TCP socket connectionwith the Robajax Web Application Gateway. This is done quite easily because the RobajaxWeb Application Gateway has a static IP address. In order to establish a TCP socketconnection, the iRobot needs to know what port the Robajax Web Application Gateway islistening on (see Figure 5.13). A SOAP Request for the port number is sent to GlassFish onport 8080 which will invoke a Web Service on the Robajax Web Application Gateway. The