# Analysis

Analysis is the final stage of the DiffBench workflow. It processes the results of completed experiments and produces aggregated tables, reports, and visualisations that make it possible to compare model performance across runs.

This stage is typically executed after one or more training experiments have completed.

For an overview of the complete workflow, see [Pipeline Overview](../overview/pipeline.md).

## Overview

During training and evaluation, DiffBench stores metrics, predictions, and experiment configurations for each run.

The analysis stage collects these outputs and transforms them into higher-level summaries.

At a high level, the workflow is:

```text
Completed experiments
        │
        ▼
Collect fold-level results
        │
        ▼
Aggregate experiments
        │
        ├── Summary tables
        ├── Dataset reports
        ├── Coverage summaries
        └── Visualisations
```

This separation allows experiments to be analyzed without rerunning model training.

---

## Inputs

The analysis stage reads the outputs generated by completed DiffBench experiments.

By default, experiments are stored under:

```text
exp_outputs/experiments/
```

with one directory per experiment:

```text
exp_outputs/
└── experiments/
    ├── exp_<run_id>/
    ├── exp_<run_id>/
    └── ...
```

Each experiment directory contains the results and configuration required by the analysis pipeline.

---

## Outputs

Analysis results are written to the experiment output directory and are generally divided into two categories:

* **summary outputs**, containing aggregated results across experiments;
* **plots**, containing visualisations for individual runs or experiment comparisons.

Typical outputs include:

```text
exp_outputs/
├── summary/
│   ├── global_metrics.parquet
│   ├── summary_metrics.parquet
│   ├── comprehensive_table.parquet
│   ├── <dataset>_report.txt
│   └── coverage_table.txt
│
└── plots/
    └── <run_id>/
```

These outputs provide different levels of aggregation, from fold-level metrics to one-row-per-experiment summaries.

For a detailed description of each generated file and its contents, see the [Results Reference](../reference/results.md).

---

## Aggregating experiment results

DiffBench can combine results from multiple experiments into common summary tables.

This makes it possible to compare:

* different models;
* datasets;
* prediction targets;
* feature representations;
* preprocessing choices;
* experimental configurations.

The analysis pipeline preserves configuration information alongside the performance metrics, allowing results to be filtered and compared according to the experiment settings.

---

## Reports and visualisations

In addition to machine-readable tables, DiffBench can generate human-readable reports and plots.

These may include:

* per-dataset performance summaries;
* comparisons between experiment configurations;
* best-performing runs;
* coverage summaries across datasets and models;
* per-run visualisations;
* learning curves.

The exact outputs generated depend on the selected analysis configuration and on the available experiment results.

---

## Running the analysis

The recommended way to run the analysis stage is through the command-line interface:

```bash
diffbenchmark-analysis
```

By default, this generates the configured summary tables and plots.

Individual parts of the analysis can be enabled or disabled through Hydra overrides.

For example, to generate tables without plots:

```bash
diffbenchmark-analysis plots=false
```

or to regenerate existing plots:

```bash
diffbenchmark-analysis force_plots=true tables=false
```

For the complete set of analysis configuration options, see the [Configuration Reference](../reference/configuration.md).

---

## Incomplete or failed experiments

By default, the analysis stage is intended to operate on completed experiment runs.

Additional debugging options can be used when inspecting experiments that are still running or that failed before completion.

These options are mainly useful during development and debugging rather than for standard benchmark analysis.

See the [Configuration Reference](../reference/configuration.md) for the corresponding settings.

---

## Relationship with evaluation

The evaluation stage computes performance metrics for individual cross-validation folds.

The analysis stage operates on those stored results and aggregates them across folds and experiments.

In other words:

```text
Predictions
    │
    ▼
Evaluation
    │
    └── Fold-level metrics
              │
              ▼
           Analysis
              │
              ├── Cross-fold summaries
              ├── Experiment comparisons
              └── Reports and plots
```

For details about how metrics are computed, see [Evaluation](evaluation.md).

For the available metrics, see the [Metrics Reference](../reference/metrics.md).

---

## Related documentation

For more detailed information:

* See the [Results Reference](../reference/results.md) for generated files, table contents, and report formats.
* See the [Configuration Reference](../reference/configuration.md) for analysis options.
* See the [Analysis API Reference](../api/analysis.md) for programmatic use of the analysis utilities.
* See [Evaluation](evaluation.md) for how fold-level metrics are produced.

---

## Next step

Analysis is the final stage of the standard DiffBench pipeline.

The generated tables, reports, and plots can then be used to compare experiments, inspect model behaviour, and summarize benchmark results.