Links

A blog about Chat bots (mainly Watson), and life in general.

Optimizing intents with no K-Fold

So as I blogged about earlier, to help find conflicting training questions in your intents you would normally use K-Fold. There are issues in using this though.

It removes some of your training data that can weaken intents.

You have to balance number of folds for accuracy over speed.

Requires to make multiple workspaces.

Once you have created the results, you have to figure them out.

Looking back at an old blog post on compound questions, I realized that this already sees where a question can be confused with different intents.

So the next step was to determine to equate this to intent testing. Removing a question and retraining is not an option. It takes too long, and offers nothing over the existing K-Fold testing.

Sending the question back as-is will always return a 100% match. Every other intent gets a 0.0 score, so you can’t see confusion. But what if you changed the question?

First up, I took the workspace from the compound questions. Despite being a tiny workspace it works quite well. So I had to manufacture some confusion. I did this by taking a question from one intent and pasting it into another (from #ALLOW_VET to #DOG_HEALTH).

Can my vet examine the puppies?

Next up for the code we define a “safe word”. This is prepended to any question when talking to Watson Assistant. In this case I selected “SIO”. What was interesting when testing this is that even if the safe word doesn’t make sense, it can still impact the results on an enterprise level (ie. 1000’s of questions).

We end up with this response:

SIO Can my vet examine the puppies?

ALLOW_VET = 0.952616596221924
DOG_HEALTH = 0.9269126892089845

Great! We can clearly see that these intents are confused with each other. So using the K-Means code from before we can document actual confusion between intents.