Evaluation & Calculate Top-N Accuracy: Top 1 and Top 5
Introduction
In machine learning, particularly in classification tasks, we often need to evaluate the performance of our models. One common metric is Top-N Accuracy, which measures how well the model predicts the top N most likely classes for a given input.
What is Top-N Accuracy?
Top-N Accuracy is a performance metric that measures the percentage of instances where the correct class is present within the top N predicted classes.
- Top-1 Accuracy: This is the simplest case, where we check if the model correctly predicts the most likely class (highest probability). It is equivalent to the standard accuracy metric.
- Top-5 Accuracy: This measures the percentage of instances where the correct class is among the top 5 most likely classes predicted by the model.
Why Use Top-N Accuracy?
- Multi-class Classification: In tasks with many classes, it’s often difficult for the model to perfectly predict the correct class every time. Top-N Accuracy provides a more lenient evaluation by considering multiple possibilities.
- Ranking and Recommendations: When dealing with ranking or recommendation systems, Top-N Accuracy is useful to assess how well the model prioritizes the correct items within a larger list.
Calculating Top-N Accuracy
Let’s illustrate with an example. Suppose we have the following predicted probabilities for a single instance with 5 classes:
| Class | Predicted Probability |
|---|---|
| Class A | 0.20 |
| Class B | 0.15 |
| Class C | 0.45 |
| Class D | 0.10 |
| Class E | 0.10 |
The correct class for this instance is Class C.
Top-1 Accuracy:
The model predicts Class C as the most likely class (highest probability), so Top-1 Accuracy is 1 (or 100%).
Top-5 Accuracy:
All 5 classes are considered in Top-5 accuracy, and the correct class (Class C) is present within the top 5. Hence, Top-5 Accuracy is also 1 (or 100%).
Code Example
Here’s a Python code snippet to calculate Top-1 and Top-5 Accuracy:
import numpy as np
def top_n_accuracy(predictions, labels, n):
"""Calculates Top-N accuracy.
Args:
predictions: A numpy array of predicted probabilities with shape (num_instances, num_classes).
labels: A numpy array of true labels with shape (num_instances).
n: The value of N for Top-N accuracy.
Returns:
The Top-N accuracy score.
"""
top_n_indices = np.argsort(predictions, axis=1)[:, -n:]
correct_predictions = np.sum(labels == top_n_indices)
return correct_predictions / len(labels)
# Example usage
predictions = np.array([[0.2, 0.15, 0.45, 0.1, 0.1],
[0.1, 0.3, 0.2, 0.4, 0],
[0.5, 0.2, 0.1, 0.1, 0.1]])
labels = np.array([2, 3, 0])
top1_acc = top_n_accuracy(predictions, labels, 1)
top5_acc = top_n_accuracy(predictions, labels, 5)
print("Top-1 Accuracy:", top1_acc)
print("Top-5 Accuracy:", top5_acc)
Output:
Top-1 Accuracy: 0.3333333333333333 Top-5 Accuracy: 1.0
Conclusion
Top-N Accuracy is a valuable metric for evaluating the performance of multi-class classification models and recommendation systems. It offers a more nuanced perspective by considering the top N predictions instead of solely focusing on the most likely class. By calculating and analyzing Top-1 and Top-5 accuracy, you can gain a better understanding of how well your model ranks the relevant classes, especially when dealing with diverse and complex datasets.