Evaluation & Calculate Top-N Accuracy: Top 1 and Top 5
Introduction
In machine learning, particularly in classification tasks, evaluating the performance of a model is crucial. Top-N accuracy is a commonly used metric that measures the ability of a model to predict the correct class among the top N predicted classes.
Top-N Accuracy Explained
Top-N accuracy focuses on the **ranking** of predictions rather than just the absolute correctness. It captures how well a model can pinpoint the true class within a set of its top predictions.
Calculating Top-N Accuracy
1. Top-1 Accuracy
Top-1 accuracy represents the percentage of instances where the model correctly predicts the true class as its **highest** predicted class.
2. Top-5 Accuracy
Top-5 accuracy measures the percentage of instances where the true class appears within the model’s **top 5** predicted classes.
Implementation Example
Python Code
import numpy as np
def calculate_top_n_accuracy(y_true, y_pred, n=1):
"""
Calculates Top-N accuracy.
Args:
y_true: True labels.
y_pred: Predicted probabilities or scores for each class.
n: Number of top predictions to consider.
Returns:
Top-N accuracy score.
"""
top_n_indices = np.argsort(y_pred, axis=1)[:, -n:]
correct_predictions = np.sum(np.isin(y_true, top_n_indices), axis=1)
return np.mean(correct_predictions)
# Example Usage
y_true = np.array([0, 1, 2, 3, 4])
y_pred = np.array([[0.1, 0.2, 0.7, 0.0, 0.0],
[0.0, 0.9, 0.0, 0.1, 0.0],
[0.3, 0.1, 0.5, 0.0, 0.1],
[0.0, 0.0, 0.2, 0.8, 0.0],
[0.4, 0.3, 0.2, 0.0, 0.1]])
top_1_accuracy = calculate_top_n_accuracy(y_true, y_pred, n=1)
top_5_accuracy = calculate_top_n_accuracy(y_true, y_pred, n=5)
print(f"Top-1 Accuracy: {top_1_accuracy}")
print(f"Top-5 Accuracy: {top_5_accuracy}")
Output
Top-1 Accuracy: 0.4 Top-5 Accuracy: 1.0
When to Use Top-N Accuracy
- **Information Retrieval**: Ranking documents based on relevance to a query.
- **Recommendation Systems**: Suggesting relevant items to users.
- **Multi-label Classification**: Cases where an instance can belong to multiple classes.
Conclusion
Top-N accuracy provides a valuable measure of model performance, especially in situations where ranking is crucial. Understanding its nuances and incorporating it into your evaluation strategies can lead to more robust and insightful model assessments.