How Bots and Scraping Affect Your Online Travel Site

Travel Brand’s Digital Marketing Officer Rob Gennaro joined Orion Cassetto, Distil Networks’ Director of Product Marketing, on stage for a joint presentation at the PhocusWright Conference. The goal of the presentation was to bring the industry audience up to speed on today’s sophisticated bots and the damage they can inflict on online travel businesses.

Starting at the ‘Bot’﹘tom

So, what is a bot? A bot is essentially an automated program that runs on the Internet. It could be a simple command line script, a browser plug-in, or a browser automation tool that accurately mimics human behavior. Depending on their intent, bots can be either benevolent or malicious. Good bots like Google, Yahoo, Baidu, and others perform helpful functions such as aiding visitors in finding your website, while bad bots perform malicious activities such as scanning for security vulnerabilities, stealing unique content, hijacking inventory and upsell opportunities, and committing fraud.

According to the 2018 Bad Bot Landscape Report by Distil Networks, 44% of traffic to travel sites comes from bots. That's right, almost half of a travel's site traffic might be coming from bots rather than from real human customers. The biggest problem with bots is that they’re enabling interaction with websites -- for good or bad -- on a massive scale. When they throw massive numbers of requests at your site, they cause slowdowns and downtime that create a negative user experience. Once they start stealing content and duplicating it elsewhere, SEO rankings are impacted (search engines penalize duplicate content). Then your human traffic numbers begin to decline, taking your revenues with them.

Almost anyone can get into the web scraping game

It’s incredibly cheap and easy to get into web scraping especially in online travel. Apps are available online for little or no investment, and it’s simple to use cloud computing to spin up multiple instances of these apps, each of which becomes another bot.

Lazy would be attackers are in luck as there is an even easier way to get into the scraping game; botnets. Instead of needing to manage Amazon instances and install software, attackers can simply rent a botnet for a few hours off of the dark web and point it at a target online travel website. These turnkey systems will then start scraping content – that’s all it takes.

High return on investment motivates would be attackers

The logic is simple. You could spend years of effort and millions of dollars building a content-rich online travel website, or you could steal it.

From airline tickets and hotel rooms, to user generated reviews and unique editorial content, regardless of the nature of a travel site, any unique content on a website could be stolen by bots. If a website is not specifically protected against web scraping, anyone can duplicate that content for next to nothing – no investment in research, infrastructure, personnel, or anything else necessary. That content can then be sold to a competitor, or even used against you steal your organic search traffic.

When presented with these options, why would attackers pay to license an API or to legitimately acquire content when they can bypass the authorization process entirely?

Price scraping hijacks sales and damages your brand

Price scraping is particularly egregious in the travel world, where even a few dollars’ difference in pricing can determine who gets a sale. Once a sale has been hijacked, the hijacker can pick up all upselling and cross selling opportunities – hotel rooms, rental cars, insurance, etc. – which can represent up to $40 of additional revenue per transaction. This represents a big chunk of the planned profit margins for many travel businesses.

Unfortunately, these unwanted middlemen also frequently cause friction with customers which affects their purchasing experience and overall satisfaction. One well publicized example is that of RyanAir, who has been in the news for its legal battles against aggregators like eDreams that are hooking into the company’s sales process without authorization. In the case with eDreams, Ryanair claims that the eDreams published low price, non-existence fares with Ryanair’s name.

This was allegedly done by putting Ryanair’s name in the URL of these fares to attract customers to the eDreams site, even though these tickets were not actually available from Ryanair. Ryanair argued that this impacted its customers’ perception of their brand because 82% of adults surveyed believed this bait-and-switch was Ryanair’s doing.

In the past, Ryanair has also complained of other poor user experiences caused by unauthorized aggregators who failed to accurately pass on important travel information such as special needs requests, web check-in information, flight updates and other essential information. When this information isn’t accurately relayed, miscommunications with customers such as missed flights ensue.

Online fraud targets travel loyalty programs

Loyalty programs, too, are a major target - mileage points quickly be converted to cash or used to purchase resaleable items like flights, cameras, even designer clothing. These loyalty programs are being attacked by bots performing what is known as a brute force attack, where an attacker uses a bot to systematically try millions of username/password combinations at victim sites until they get lucky and find one that works. As we all know, many people use the same credentials for multiple sites, so when a site like Ashley Madison gets hacked and 32 million credentials are suddenly available – well, you get the picture. To make matters worse, older mileage programs that use only a four-digit PIN are particularly vulnerable – that’s only 10,000 possible combinations, and a botnet can crack that in less than a minute.

Bots hit core business drivers

Like many online businesses, online travel sites are driven by customer loyalty that’s continually reinforced by predictable and high-quality customer experiences, and by customer satisfaction. Successful online business are also adept at capturing the opportunity to insert relevant upsell opportunities at the time customers are ready to buy. Between these business drivers as well as fraud, negative SEO, brand impact, loss of customer loyalty, and the cost of the web infrastructure that’s serving only these bots, it’s clear that these automated miscreants chip away at profit margins and resources of online travel sites.

Fighting back against the bots

Rob Gennaro at Red Tag has been on the receiving end of every possible bad bot experience. He’s seen slowdowns lasting up to an hour a day during peak times, while visitors got more and more frustrated, unable to ask questions or complete transactions. He’s seen countless instances of online fraud, and bots that can rip off his entire 300,000 page website in 15 minutes.

Red Tag’s engineers developed applications to block bots at the IP level, but the bots were cycling through IP addresses faster than the software could detect and block them, and the customer experience got worse and worse. Other home-grown solutions simply could not scale to Red Tag’s traffic levels, and caused as many site slowdowns as the bots themselves. It was only when Rob connected with Distil Networks that he saw how a new thought process and self-optimizing technology built on community-based intelligence, could really make a difference.

Join the conversation on Quora

In addition to writing this blog post, I’ve also answered a related question on Quora. Feel free to read my answer there and join in the conversation. I’d be happy to answer any specific questions you have!

Want to dig deeper?

Check out this webinar we hosted with Skift for more in-depth info on bots and the travel industry; you can also learn more about how Red Tag fought back against bots in this case study.

About the Author

Orion Cassetto joined Distil Networks as Director of Product Marketing in 2015, bringing with him nearly a decade of experience in the Cyber Security industry. His strengths include competitive strategy, positioning, and messaging for web application security and SaaS-based security solutions.

Comparisons

Distil

Contact

Distil Networks protects mission-critical websites, mobile apps, and APIs from automated threats without affecting the flow of business-critical traffic. We defend customers against web scraping, account takeover, transaction fraud, denial of service, competitive data mining, unauthorized vulnerability scans, spam, click fraud, and web and mobile API abuse. Only Distil’s unique, more holistic approach provides the vigilant service, superior technology, and industry expertise needed for full visibility and control over human, good bot, and bad bot traffic. As their ally in the war against bots, we provide customers with vigilant and dedicated support so that when they’re under attack, there is a team of experts ready to help. With Distil, there is finally a defense against automated attacks that is as adaptable and vigilant as the threat itself.

Under Attack? We can help with an expedited response!

By continuing to browse this or by clicking “Accept Cookies,” you agree to the storing of first- and third-party cookies on your device for the reasons specified in our Cookie Policy.