Evaluation & Calculate Top-N Accuracy: Top 1 and Top 5
Introduction
Top-N accuracy is a common metric used to evaluate the performance of ranking and recommendation systems. It measures the percentage of times the correct item is ranked within the top N positions of the predicted list. This article focuses on the calculation of Top-1 and Top-5 accuracy, two popular variations of this metric.
Top-1 Accuracy
Top-1 accuracy, also known as the “hit rate,” measures the percentage of predictions where the correct item is ranked first. It represents the system’s ability to consistently predict the most relevant item.
Calculation
To calculate Top-1 accuracy, we follow these steps:
- For each data point, determine the true relevant item.
- Generate a ranked list of predicted items.
- If the true relevant item is ranked first in the predicted list, increment the count of correct predictions.
- Divide the count of correct predictions by the total number of data points.
Code Example
def calculate_top1_accuracy(predictions, true_labels): correct_predictions = 0 for i in range(len(predictions)): if predictions[i][0] == true_labels[i]: correct_predictions += 1 return correct_predictions / len(predictions) # Example usage predictions = [[1, 2, 3], [3, 1, 2], [2, 1, 3]] true_labels = [1, 3, 2] top1_accuracy = calculate_top1_accuracy(predictions, true_labels) print(f"Top-1 Accuracy: {top1_accuracy}") Output
Top-1 Accuracy: 0.3333333333333333Top-5 Accuracy
Top-5 accuracy measures the percentage of predictions where the correct item is ranked within the top five positions of the predicted list. It provides a broader perspective on the system’s overall performance, considering the top five most likely recommendations.
Calculation
To calculate Top-5 accuracy, we follow similar steps to Top-1 accuracy but consider the top five predictions:
- For each data point, determine the true relevant item.
- Generate a ranked list of predicted items.
- If the true relevant item is ranked within the top five positions in the predicted list, increment the count of correct predictions.
- Divide the count of correct predictions by the total number of data points.
Code Example
def calculate_top5_accuracy(predictions, true_labels): correct_predictions = 0 for i in range(len(predictions)): if true_labels[i] in predictions[i][:5]: correct_predictions += 1 return correct_predictions / len(predictions) # Example usage predictions = [[1, 2, 3], [3, 1, 2], [2, 1, 3]] true_labels = [1, 3, 2] top5_accuracy = calculate_top5_accuracy(predictions, true_labels) print(f"Top-5 Accuracy: {top5_accuracy}") Output
Top-5 Accuracy: 1.0Conclusion
Top-1 and Top-5 accuracy are essential metrics for evaluating ranking and recommendation systems. Top-1 accuracy provides a measure of the system’s ability to predict the most relevant item, while Top-5 accuracy considers the top five recommendations, offering a broader evaluation of performance. By understanding and calculating these metrics, developers can gain valuable insights into the effectiveness of their models and make informed decisions to improve them.