Crossbeam and Spirent recently partnered with EANTC, an internationally recognized test center, and Heavy Reading analyst Gabriel Brown to define and emulate the most realistic real-world conditions within mobile networks, and evaluate the performance of Crossbeam X80-S as a network security device under these demanding conditions.

We agreed on one key requirement — we wanted to emulate, as closely as possible, what happens on a mobile network when subscribers are actually using their smartphones and tablets, not just a limited best-case viewpoint. Throughout the planning process, some of the questions we wanted to examine included:

How do you evaluate a "super-class" platform like the Crossbeam X80-S?

What would be meaningful to address the needs of a mobile operator/service cloud provider?

How consistent will the subscriber Quality of Experience and happiness remain when running over a hundred of gigabits per second of throughput?

Will it take 30 seconds to load a web page?

We set up the test to stress the Crossbeam platform with a true worst-case scenario, and evaluate all vectors simultaneously. We knew that the performance and quality levels created in this scenario would represent the most demanding conditions required to actually deliver service, which would enable mobile operators to accurately predict the peak performance in their networks and the devices they choose.

Simulating Real-World Mobile Users

To be realistic we wanted to fully emulate user behavior on a mobile device using the SimUser functionality on the Spirent Avalanche. For example, a mobile user will not judge the quality of the service by a single transaction (a GET). Instead, a user is more concerned with how fast the page renders on their mobile device. In our test, we wanted to form a mobile page by combining an index.html document with sub-objects, and then measure the stack (“New Reno” TCP) from bandwidth through page load time. Because we are directly measuring the render time of the mobile browser, we decided to add additional realism by emulating a tabbed mobile browser, viewing three pages with eight seconds of “browsing time by the user” per page.

We used 8 Avalanche 3100, with 4 generating traffic, and 4 as clients receiving the traffic, as follows:

Simulating Mobile Traffic

We wanted to emulate this mobile traffic as a series of transactions. For instance, depending on the actions each subscriber took, the relationship of Simulated Users (SimUser) to TCP Connections to Layer 7 Transactions varied (as would a real subscriber in a production network). So we used a distribution of 95% HTTP, 2% POP3, 1% SMTP, 1% DNS, and 1% large bulk transfer (over HTTP). Each of these subscribers generated mobile data traffic as the test ramped according to the loading curve defined in the Spirent Avalanche, increased up for a sustained period, and then ramped down. Thus each Subscriber performed the following:

Traffic

Type

Traffic Generated per SimUser
(1 Million concurrently active)

# Open TCP Connections

Transaction Count

Notes/Ending sequences

95%

HTTP

12

15
(Layer 7)

Closed all open TCP connections with a RST, the Server responded with a RST, ACK and resources deallocated

2%

POP3

1 – creating a 64k byte message to 2 users

1 - sent to mail server

1%

SMTP

Emulated POP3 server 1 or 2 64k messages

1 - each subscriber logs into the POP3 mailbox

Checked and retrieved messages.

1%

DNS

1 – subscriber issuing 1 DNS record lookup to an emulated DNS Zone with an A Record resolving to a URL

1

1%

Bulk/OS Update

1 – the subscriber requesting a 2Mb object

1

Results

The Avalanche system measured subscriber QoE bandwidth, TCP connection count and rate, transaction count and rate, errors, HTTP status codes, and more, all time trended. Over 700 metrics were read by the system every four seconds. We tried various combinations on the Crosbeam X80-S, including: baseline testing, firewall with IPS enabled, firewall with NAT enabled, and finally firewall with NAT+IPS simultaneously enabled. Across these we impressively saw 106-108 Gbps of throughput, while simultaneously maintaining 270,000 concurrent open connections per second and 675,000 transactions per second, with only tens of nanoseconds of impact on the Web browser page render time. Check out Crossbeam’s press release for details of the results as well as this whitepaper from Heavy Reading that examines the EANTC test results.