Metrics Reference#

DiffBench evaluates predictive models using task-specific performance metrics.

The metric used in an experiment depends on whether the prediction task is a classification or regression problem. Metrics are computed independently for each cross-validation fold and can later be aggregated across folds during the analysis stage.

For an overview of the evaluation workflow, see Evaluation.

Overview#

DiffBench currently distinguishes between two main types of prediction tasks:

Task

Main metric

Better value

Binary classification

Balanced accuracy

Higher

Regression

R² score

Higher

These metrics are used to compare models under the same cross-validation procedure.

At a high level:

Model predictions
        │
        ▼
Prediction task
        │
        ├── Classification ──► Balanced accuracy
        │
        └── Regression ──────► R²
        │
        ▼
Fold-level metric
        │
        ▼
Aggregation across folds

The aggregation and comparison of metrics across experiments is handled by the Analysis stage.

Classification metrics#

Balanced accuracy#

Balanced accuracy is the main classification metric used by DiffBench.

It measures classification performance while giving equal importance to each class.

For binary classification, it is defined as the average of the recall obtained for the two classes:

Balanced Accuracy = (Sensitivity + Specificity) / 2

where:

  • Sensitivity measures the proportion of positive samples correctly classified;

  • Specificity measures the proportion of negative samples correctly classified.

Balanced accuracy ranges from 0 to 1.

Value

Interpretation

1.0

Perfect classification

0.5

Chance-level performance for a balanced binary problem

< 0.5

Performance below chance

0.0

Predictions are completely incorrect

Why balanced accuracy?#

Standard accuracy can give misleading results when the target classes are imbalanced.

Consider a dataset containing:

90 controls
10 patients

A model predicting every participant as a control would obtain:

Accuracy = 90%

despite completely failing to identify patients.

Balanced accuracy evaluates performance independently for each class before averaging them, making it more appropriate when the number of samples per class differs.

Example#

Suppose a model obtains:

Sensitivity = 0.80
Specificity = 0.90

Then:

Balanced Accuracy = (0.80 + 0.90) / 2
                  = 0.85

The resulting balanced accuracy is therefore 0.85.

In DiffBench#

Balanced accuracy is computed independently for each cross-validation fold.

For example:

Fold 1 ──► 0.81
Fold 2 ──► 0.84
Fold 3 ──► 0.79
Fold 4 ──► 0.83
Fold 5 ──► 0.82
              │
              ▼
       Mean ± variability

The fold-level values are preserved in the experiment outputs, while their aggregated statistics are generated during analysis.

See the Results Reference for the corresponding output files.

Regression metrics#

R² score#

The coefficient of determination, commonly referred to as R², is the main regression metric used by DiffBench.

R² measures how well model predictions explain the variability of the target variable.

It is defined as:

           Σ (yᵢ - ŷᵢ)²
R² = 1 - ───────────────
           Σ (yᵢ - ȳ)²

where:

  • yᵢ is the true target value;

  • ŷᵢ is the predicted value;

  • ȳ is the mean of the true target values.

Interpretation#

Unlike many performance metrics, R² is not restricted to the interval [0, 1].

R²

Interpretation

1.0

Perfect predictions

0.0

Equivalent to always predicting the mean of the target

< 0.0

Worse than predicting the target mean

For example:

R² = 0.70

means that the model explains approximately 70% of the variance in the target values for the evaluated data.

A negative value is possible:

R² = -0.20

This indicates that the predictions perform worse than a simple baseline that always predicts the mean of the target.

In DiffBench#

R² is computed independently for each cross-validation fold:

Fold 1 ──► R²
Fold 2 ──► R²
Fold 3 ──► R²
Fold 4 ──► R²
Fold 5 ──► R²
              │
              ▼
       Mean ± variability

As with classification metrics, fold-level results are stored by the evaluation pipeline and aggregated during analysis.

Baseline models and metric interpretation#

Performance metrics should generally be interpreted relative to an appropriate baseline.

DiffBench includes baseline models that provide a reference against which predictive models can be compared.

For classification, a dummy classifier can represent a simple prediction strategy that does not learn meaningful relationships between the imaging features and target.

For regression, a dummy regressor can provide a baseline based on a simple target statistic such as the training-set mean.

The purpose of these models is not to obtain competitive performance, but to verify that more complex models provide useful predictive information beyond trivial prediction strategies.

For the available baseline models, see the Models Reference.

Metrics and cross-validation#

DiffBench computes evaluation metrics separately for each cross-validation fold rather than calculating a single metric from the complete dataset.

For example:

                    Dataset
                       │
                       ▼
                Cross-validation
                       │
            ┌──────────┼──────────┐
            ▼          ▼          ▼
          Fold 1     Fold 2      ...
            │          │
            ▼          ▼
         Predict     Predict
            │          │
            ▼          ▼
         Metric      Metric
            └──────────┬──────────┘
                       ▼
                 Aggregation

This makes it possible to inspect both:

  • average predictive performance;

  • variability in performance across data splits.

The analysis stage produces the corresponding cross-fold summaries.

See Analysis for details.

Fold-level and aggregated metrics#

It is useful to distinguish between evaluation and analysis in DiffBench.

During evaluation:

predictions
    │
    ▼
metric computation
    │
    ▼
fold-level metrics

During analysis:

fold-level metrics
       │
       ▼
aggregation across folds
       │
       ▼
mean / variability
       │
       ▼
experiment comparisons

This separation preserves the individual fold results while also providing concise summaries for comparing experiments.

The generated metric tables are described in the Results Reference.

Choosing the appropriate metric#

The evaluation metric must correspond to the prediction task.

For a binary classification task:

task = classification
        │
        ▼
Balanced accuracy

For a continuous prediction task:

task = regression
        │
        ▼
R²

Experiments with different prediction tasks should not be compared directly using their raw metric values.

For example, a balanced accuracy of 0.80 and an R² of 0.80 represent fundamentally different quantities.