Chatbot Analytics 101: The Essential Metrics You Need to Track

Why Chatbot Analytics Matter

Analytics are often overlooked and underappreciated when it comes to chatbots. While chatbot analytics are unlikely to make or break the success of a chatbot, they can provide valuable insight into opportunities for growth and improvement by allowing chatbot builders to get into the minds of users. The KPIs (Key Performance Indicators) that you need to track will often vary based on the use case of the chatbot and the demographics of the user base; however, several key metrics will provide valuable insight for just about any chatbot.

Active Users

Monitoring active users is a must for most software applications, and chatbots are no different. This KPI allows you to get a feel for the overall popularity of your chatbot and is a good barometer of its success. If this metric is trending downward, it could be an indicator that you need to rethink the use cases of your chatbot and its design. This is a simple yet powerful metric to include in any chatbot analytics.

For even more insight, you can monitor the recurring active users of your chatbot to get a feel for how often users are coming back to user your chatbot after the initial use. Poor performance in regards to recurring active users could be a sign of high dissatisfaction rates amongst first-time users.

Session Length / Steps Per Conversation

These KPIs are critical to assessing the effectiveness of your chatbot regarding its ability to carry on a meaningful conversation with users. While the ideal session length will vary based on its use cases and the context of the conversation, short session lengths are often indicative of some form of failure unless your chatbot can resolve user inquiries almost immediately. Make sure to use some sort of timeout, so that session lengths are not inflated by idle periods.

The number of steps per conversation is another metric that you need to set a target for and monitor. Ideally, most chatbots should aim to resolve a user’s inquiry in as few conversation steps (a conversation step is one back-and-forth message exchange between a user and a chatbot) as possible.

Impatient users will leave a chatbot conversation if they have to go through too many conversation steps to reach the value they’re looking for. On the flipside, conversations with very few conversation steps are likely to indicate glaring chatbot flaws that are causing users to lose faith early on. Set a good impression early on in the conversation to keep users engaged and active with your chatbot. An effective chatbot welcome message is a great way to accomplish this.

User Chatbot Ratings

Allowing users to rate your chatbot is an exceptional method of providing users with the opportunity to express satisfaction or dissatisfaction with your chatbot. In particular, it is extremely valuable to get this feedback on a per chatbot message basis rather than on a per chatbot basis as you will be able to better identify the weak points in your chatbot’s conversation flow. Of course, poor ratings are going to be indicative of flaws that are leaving users dissatisfied. Examples of these flaws include poor conversation design, incorrect answers, knowledge gaps, and repetitive responses.

Chatbot Fallback Responses

Chatbots invoke fallback responses when they’re unable to find a proper response to a user’s message. Instead of saying nothing, it is better for a chatbot to respond by letting the user know that a match wasn’t found. Monitoring how often this is occurring and the user messages that are invoking fallback responses will help you to be able to identify knowledge gaps, faulty Natural Language Processing (NLP), and unclear expectations from the end users in regards to what the chatbot should/shouldn’t know.

If users are frequently asking for something that your chatbot doesn’t know, then you should either look to fill this knowledge gap or make it explicitly clear that the chatbot can’t provide this value. On the other hand, if users are frequently getting your chatbot’s fallback response when asking for something that your chatbot does know then this is an indication that you may need to train your chatbot’s NLP better to recognize all of the variances in which users can phrase the inquiry.

Which Chatbot Analytics Platform Should You Use?

To successfully analyze the mentioned metrics you will need to utilize a chatbot analytics platform. Fortunately, a lot of chatbot solutions come with their own integrated set of analytics for you to use. If your chatbot solution is lacking in regards to analytics, then you can try to utilize a 3rd-party chatbot analytics solution. Chatbase and dashbot are two of the more popular 3rd-party chatbot analytics platforms on the market.

In the screenshot below, you can see a report available via Chatbase’s chatbot analytics that allows you to see where conversational traffic is flowing, users satisfaction or dissatisfaction at specific steps in the conversation, and the rate of user dropoff at each stage of the conversation flow.

Ideally, you may prefer to use a chatbot platform that has its own built-in analytics, so you don’t have to go through the hassle of integrating and setting up analytics through a 3rd-party service such as Chatbase. While these services tout their ease-of-use, for those that aren’t technically savvy the setup and integration process could be demanding. Regardless, thanks to these 3rd-party chatbot analytics platforms you can rest easy knowing that you will always have options when it comes to your chatbot analytics needs.

So what’s next?

The Relay Chatbots have advanced Natural Language Processing (NLP) capabilities that will not only wow your customers but greatly improve your team’s ability to provide a superior customer service or support experience.

We would love to give you a demo to show you how powerful our native chatbots are when combined with the Relay intelligent support platform. Schedule a demo, and we’ll help you transform your support and service operations.

Want to read more about how AI will impact the future of customer service and support? Sign up for our monthly newsletter that shares our best content for free and follow our publication on Medium!