Operation Break the Bots: How Customer Support Chatbots Fail

Facebook’s bot fail was well-publicized. Customer support chatbots embedded in Facebook messenger were unable to communicate with human users seven out of ten times. But even so, plenty of companies still overly rely on bots to handle their interactions with customers. As avid believers that a human plus machine approach is the best approach for companies heavily invested in their brands and customers, we decided to interact with some customer support chatbots ourselves to gain insight into how and why bots fail.

Our awesome sales leader and health enthusiast, Vanessa, went head to head with the Whole Foods bot on Facebook Messenger. Despite making clear and concise inquiries, at the end of her conversation with the Whole Foods bot Vanessa was still hungry with no clue how to find the closest salad bar. Despite entering her Venice zip code, the bot failed to provide the address of the Venice, California Whole Foods, which was geographically closest to her.

Megan, our Vice President of Customer Success experienced a similar snafu with the Sephora Assistant messenger bot. Responding to an automated menu provided by the bot, Megan communicated that she wanted to book a makeover. At that point, the Sephora Assistant responded with a friendly message (rather than an automated menu) greeting Megan and asking her which location she would like to book. Megan responded with a friendly, “Hi there!” but before she was able to follow up with her zip code, the bot reverted to another automated menu.

What went wrong? Since the bot was expecting the architecture of Megan’s response to be numeric, but she instead responded with a string of text, the bot routed her to a default menu provided when a customer’s inquiry deviates from the script.

Our Creative Director, Meredith, chose to engage Nanorep, one of the bots included in Forrester’s 2017 list Top 10 Chatbots For Enterprise Customer Service. She found that Nanorep had informative canned responses about its customer service capabilities, but had a very hard time keeping up when she deviated from the script.

Tony, CEO of RapportBoost.AI, contacted the Ebay bot through Facebook Messenger. As CEO of an AI company, Tony has plenty of experience testing and interacting with bots online. He was pleased to find that theEbay bot on Facebook Messenger wasactually a live chat agent, and said so immediately. After making the human connection, Ebay provided Tony with an external link to access their Personal Assistant bot should he want to shop using an assistive, automated system.

Karim, our Sr. Director of Science and Engineering, faced off with Botsify. With over twenty years experience in business technology, it comes as no surprise that Karim immediately outsmarted his bot. Inferring that the bot was a sales representative for Botsify, Karim asked some basic questions about the company. This customer support chatbot was unable to provide information about the Botsify product, instead deferring to a menu allowing him to report an issue. Karim’s inquiry brings up an interesting point. Unless explicitly told the experience level or subjects of expertise of a bot, there’s no way for a customer to know what they can and can’t ask until it’s too late.

Finally, our Chief Data Scientist, Dr. Michael Housman, chose to interact with the 1-800 Flowers assistant on Facebook Messenger. Similar to the Ebay bot, Michael was immediately given the option to continue his interaction with a human customer service representative before proceeding. The 1-800 Flowers assistant prompted him to choose an occasion from a set list and then prompted him to choose from another list of arrangements. After browsing the selection, Michael inquired about adding carnations, which weren’t included in the list. The bot was unable to pull up a list of arrangements with carnations, instead, prompting him to choose from the options provided.

Our competition ended with a landslide victory for RapportBoost.AI, but winning didn’t feel so good. Why? Because our win was the summary of a handful of failed customer service interactions that made us lose confidence in some of our favorite brands.

There’s a big gap between the AI portrayed in the media and the AI products available for enterprise use. We hear about Google’s AI computer, AlphaGo, beating World Champion Lee Sedol at the ancient Chinese strategy game, Go. Or about Microsoft’s AI computer, Watson, beating Jeopardy’s greatest champions. There were even stories that attributed Facebook’s removal of messenger bots to their hyper-intelligence when they were really nixed because of their inability to hold conversations with humans. But the reality is that the programming and computing power required for these super-intelligent machines far surpasses the resources of most companies. Customer support chatbots use similar AI applications including machine learning and natural language processing, but on a much smaller scale.

About Tony Medrano

Tony started his career as a technology entrepreneur out of his Stanford University dorm as co-founder and President of DoDots, the first desktop-to-mobile app platform. He built the company to 100+ employees, raised $25M in venture capital and managed all company operations, personnel and deals.
He was most recently CEO of Boopsie, a mobile Platform-as-a-Service company that he grew for 4 years and successfully sold to a strategic partner in 2015. Tony has also served as Vice President of Sales / Business Development at Reply! and SmartDrive Systems.
Tony has dedicated his career in technology to leading sales teams and understands the need for automation tools that help sales reps increase efficiency, close more business and increase margins.
Tony received his MBA and JD from Stanford, M.A. from Columbia and B.A. from Harvard.