How is model accuracy calculated?

Monitoring model and tag accuracy ensures your models and, therefore, your automations are performing as well as possible.

Prerequisites

In order to view model and tag accuracy, you need first to have:

• A classification model (model accuracy is not yet supported for entity extraction models).
• Correct filled in feedback rules.

About the confidence level

In order to make sure the model and tag accuracy is as accurate as possible, a confidence level is used. This confidence level is based on the amount of feedback that was received for a certain tag (with feedback being the correction or verification of a tag prediction).

Confidence = Messages with feedback / All messages

Example: Tag is predicted 1000 times, and 500 times the prediction is followed with feedback. Confidence level = 50%.

Important to note is that there is also an absolute minimum of feedback that is needed:

• For model accuracy: 100 messages with feedback.
• For tag accuracy: 10 messages with feedback.

The time period for including feedback in the confidence calculation is the past 30 days or until the last training date if this is less than 30 days ago.

About the accuracy percentage

Accuracy percentage is calculated as an F1-score, which is a combination of Recall and Precision percentage.

Recall = Correct messages with tag / All messages with tag

“Of all messages that should have this tag, what percentage did we correctly identify?”

• 1000 messages with feedback tag X

• Of which 800 correctly predicted

→ 800 correct messages / 1000 messages = 0.80 (80%)

Precision = Correct predictions with tag / All predictions with tag

“Of all messages we tagged with this label, what percentage were correct?”

• 1000 predictions tag X

• Of which 800 have correct feedback

→ 800 correct predictions / 1000 predictions = 0.80 (80%)

F1 = 2 * (Recall * Precision) / (Recall + Precision)

About tag confusion

Within the side panel of a model or tag it is possible to see exactly how tags are mispredicted using the tag confusion. This shows on the left the original tag that was predicted and on the right the correct tag, together with the frequency of this confusion happening.

Note that confusion follows the same time window as confidence, the past 30 days or until the last training date if this is less than 30 days ago.