Demoren/nes-surrogate-architectures

NES Surrogate Dataset

Overview

This dataset contains trained neural architectures, their predictions, and validation performance, designed for studying:

• —surrogate modeling of neural architectures
• —diversity estimation between models
• —ensemble construction strategies

Each architecture is associated with:

• —its structure (DARTS-like cell)
• —model weights
• —validation predictions
• —validation accuracy

Dataset Structure

CIFAR10/
CIFAR100/
FashionMNIST/

Each dataset directory contains:

architectures/
weights/

architectures/

Each JSON file contains:

• —architecture definition (DARTS-like DAG)
• —validation predictions
• —validation accuracy

Predictions are computed on a shared validation split, enabling construction of pairwise similarity matrices.

weights/

Contains trained model weights corresponding to each architecture in safetensors format.

Example Entry

{
  "architecture": {
    "normal/op_2_0": "sep_conv_3x3",
    "normal/input_2_0": [1],
    "normal/op_2_1": "sep_conv_5x5",
    "normal/input_2_1": [0]
  },
  "valid_predictions": [6, 2, 5, 6, 3],
  "valid_accuracy": 0.76
}

Dataset Size

• —~3,000 models per dataset
• —Total: architectures, predictions, and weights for three benchmarks

Data Split

For each dataset, the original training set is split into:

• —20% training subset
• —80% validation subset

The split is performed using:

• —fixed random seed: 42
• —torch.utils.data.Subset

The validation subset is used to:

• —compute model accuracy
• —generate prediction vectors for diversity estimation

Training Setup

Architectures are trained under a reduced configuration to limit computational cost:

• —optimizer: SGD
• —learning rate: cosine schedule from 0.025 → 1e-3
• —weight decay: 3e-4
• —batch size: 96
• —auxiliary loss weight: 0.4

Training Statistics

Note: Models are not trained to full convergence. They are trained for a fixed number of epochs sufficient to obtain reliable relative performance estimates.

Key Properties

• —DARTS-like architecture search space
• —Graph-based representation (DAGs)
• —Aligned predictions across models
• —Supports diversity estimation via prediction similarity
• —Suitable for surrogate-based ranking and selection

Intended Use

This dataset enables:

• —training accuracy surrogate models
• —learning diversity embeddings (e.g., via triplet loss)
• —constructing similarity matrices between models
• —analyzing relationships between architecture and predictions

Example similarity metric:

similarity = (y_i == y_j).mean()

Relation to Paper

This dataset accompanies the paper:

"Surrogate Assisted Diversity Estimation in Neural Ensemble Search"

It is used to:

• —train surrogate models for accuracy and diversity
• —guide ensemble construction
• —study scaling behavior with respect to dataset size

Notes

• —Similarity between models is computed from prediction agreement
• —The dataset is designed for relative ranking, not absolute accuracy
• —Can be used with alternative diversity metrics (e.g., correlation, divergence)