Evaluation & Calculate Top-N Accuracy: Top 1 and Top 5

Evaluation & Calculate Top-N Accuracy: Top 1 and Top 5

Introduction

In machine learning, particularly in classification tasks, evaluating the performance of a model is crucial. Top-N accuracy is a common evaluation metric that measures how well a model predicts the top N most likely classes for a given input. This article will delve into the concepts of Top-1 and Top-5 accuracy, their calculation, and practical examples.

Top-1 Accuracy

Top-1 accuracy measures the percentage of instances where the model correctly predicts the most likely class. It is often used as the primary evaluation metric in classification tasks where the goal is to predict a single, definitive class.

Top-5 Accuracy

Top-5 accuracy expands on Top-1 by considering the top 5 most likely classes predicted by the model. It measures the percentage of instances where the true class is among the top 5 predictions. This metric is particularly relevant when there are multiple possible correct classes for an input, or when the model’s confidence in its predictions is not very strong.

Calculation

To calculate Top-N accuracy, we need to consider the following:

• Predicted Classes: The model’s output, typically a list of predicted probabilities for each class.
• True Class: The actual class label for the input.

Steps:

1. For each input, rank the predicted classes based on their probabilities in descending order.
2. Check if the true class is present within the top N predicted classes.
3. Calculate the ratio of instances where the true class is among the top N predictions to the total number of instances.

Example

In this example, the Top-1 accuracy is 2/3 (66.67%) and the Top-5 accuracy is 3/3 (100%).

Code Implementation

Here’s a Python code snippet demonstrating the calculation of Top-1 and Top-5 accuracy:

import numpy as np

def calculate_top_n_accuracy(predictions, true_labels, n):
  """Calculates Top-N accuracy for a given set of predictions and true labels.

  Args:
    predictions: A numpy array of shape (num_samples, num_classes) containing
        the predicted probabilities for each class.
    true_labels: A numpy array of shape (num_samples,) containing the true class
        labels.
    n: The value of N for Top-N accuracy calculation.

  Returns:
    The Top-N accuracy as a float.
  """
  top_n_indices = np.argsort(predictions, axis=1)[:, -n:]
  correct_predictions = np.sum(np.isin(true_labels, top_n_indices), axis=1)
  top_n_accuracy = np.mean(correct_predictions)
  return top_n_accuracy

# Example usage:
predictions = np.array([[0.2, 0.6, 0.1, 0.1],
                      [0.1, 0.3, 0.5, 0.1],
                      [0.4, 0.2, 0.3, 0.1]])
true_labels = np.array([1, 2, 0])

top_1_accuracy = calculate_top_n_accuracy(predictions, true_labels, 1)
top_5_accuracy = calculate_top_n_accuracy(predictions, true_labels, 5)

print("Top-1 Accuracy:", top_1_accuracy)
print("Top-5 Accuracy:", top_5_accuracy)

Output:

Top-1 Accuracy: 0.3333333333333333
Top-5 Accuracy: 1.0

Conclusion

Top-N accuracy is a valuable evaluation metric for classification tasks, providing insights into the model’s ability to predict the most likely classes. Top-1 accuracy focuses on the single most likely class, while Top-5 accuracy considers a broader range of potential predictions. Choosing the appropriate Top-N metric depends on the specific requirements of the task and the model’s confidence in its predictions.